Custom rack for scalable position tracking system

ABSTRACT

A rack for a scalable tracking system includes weight sensors disposed on shelves that hold items. The weight sensors detect the weight of the items and communicates signals indicating that weight to a circuit board positioned in the rack. The circuit board communicates these detected weights to a weight server that determines, based on these weights, whether items were removed from the shelves.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/663,669 filed Oct. 25, 2020, by Deepanjan Paul et al., and entitled“CUSTOM RACK FOR SCALABLE POSITION TRACKING SYSTEM,” which isincorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to a distributed system for trackingphysical positions of people and objects.

BACKGROUND

Position tracking systems are used to track the physical positions ofpeople and/or objects.

SUMMARY OF THE DISCLOSURE

Position tracking systems are used to track the physical positions ofpeople and/or objects in a physical space (e.g., a store). These systemstypically use a sensor (e.g., a camera) to detect the presence of aperson and/or object and a computer to determine the physical positionof the person and/or object based on signals from the sensor. In a storesetting, other types of sensors can be installed to track the movementof inventory within the store. For example, weight sensors can beinstalled on racks and shelves to determine when items have been removedfrom those racks and shelves. By tracking both the positions of personsin a store and when items have been removed from shelves, it is possiblefor the computer to determine which user in the store removed the itemand to charge that user for the item without needing to ring up the itemat a register. In other words, the person can walk into the store, takeitems, and leave the store without stopping for the conventionalcheckout process.

For larger physical spaces (e.g., convenience stores and grocerystores), additional sensors can be installed throughout the space totrack the position of people and/or objects as they move about thespace. For example, additional cameras can be added to track positionsin the larger space and additional weight sensors can be added to trackadditional items and shelves. There is a limit, however, to the numberof sensors that can be added before the computing capabilities of thecomputer are reached. As a result, the computing power of the computerlimits the coverage area of the tracking system.

One way to scale these systems to handle larger spaces is to addadditional computers and to divide the sensors amongst these computerssuch that each computer processes signals from a subset of the sensors.However, dividing the sensors amongst multiple computers introducessynchronization issues. For example, the sensors may not communicatesignals to their respective computers at the same time orsimultaneously. As another example, sensors may have different latencieswith their respective computers and thus, it may take more time forsignals from one sensor to reach a computer than signals from anothersensor. As a result, the sensors and computers become desynchronizedwith each other and it becomes more difficult for the computers todetermine, in a cohesive way, the position of persons or objects in thespace and when items were removed.

This disclosure contemplates an unconventional, distributed trackingsystem that can scale to handle larger spaces. The system uses an arrayof cameras, multiple camera clients, a camera server, weight sensors, aweight server, and a central server to determine which person in thespace took an item and should be charged for the item. The cameraclients each process frames of videos from a different subset of camerasof the array of cameras. Each camera client determines coordinates forpeople detected in the frames and then timestamps these coordinatesbased on when the frames were received by the camera client. The cameraclients then communicate the coordinates and timestamps to a cameraserver that is responsible for coordinating the information from thecamera clients. The camera server determines, based on the coordinatesand timestamps from the camera clients, the positions of people in thespace. The weight server processes signals from the weight sensors todetermine when items were removed from shelves in the space. The centralserver uses the positions of people in the space from the camera serverand the determinations from the weight server of when items were removedfrom shelves to determine which people in the space took which items andshould therefore be charged.

Generally, the camera server protects against desynchronization byassigning the coordinates from the multiple camera clients to windows oftime based on the timestamps. The camera server then processes thecoordinates assigned to a particular time window to determine overallcoordinates for people in the space during that time window. Theduration of the time window can be set to be larger than thedesynchronization that is expected to occur to mitigate the effects ofdesynchronization. For example, if the cameras and camera clients areexpected to desynchronize by a few milliseconds, then the time windowcan be set to last 100 milliseconds to counteract the desynchronization.In this manner, the number of cameras and camera clients can beincreased to scale the system to handle any suitable space.

This disclosure also contemplates an unconventional way of wiringcameras in the array of cameras to the camera clients. The cameras arearranged as a rectangular grid above the space. Each camera in the gridis wired to a particular camera client according to certain rules. Forexample, no two cameras that are directly adjacent to one another in thesame row or column of the grid are wired to the same camera client. Asanother example, cameras that are arranged along a diagonal in the gridare wired to the same camera client. In this manner, a small area of thegrid should include cameras that are wired to each and every cameraclient in the system. As a result, even if one camera client were to gooffline (e.g., maintenance, error, or crash), there would still beenough coverage from the remaining camera clients to track the positionsof people in the small area. Thus, this arrangement of the camerasimproves the resiliency of the system.

This disclosure further contemplates an unconventional rack and shelfdesign that integrates weight sensors for tracking when items have beenremoved from the racks and shelves. Generally, the rack includes a base,a vertical panel, and a shelf. The base forms an enclosed space in whicha printed circuit board is positioned, and the base includes a drawerthat opens to provide access to the enclosed space and the circuitboard. The vertical panel is attached to the base, and the shelf isattached to the vertical panel. Weight sensors are positioned within theshelf. The base, panel, and shelf each define a cavity. The cavity inthe shelf and the cavity in the panel are at least partially aligned.Each weight sensor communicates signals to the printed circuit boardthrough a wire that runs from that weight sensor, through the cavity ofthe shelf, the cavity of the panel, and the cavity of the base, to thecircuit board.

Certain embodiments include an unconventional tracking system thatincludes separate components (e.g., camera clients, camera servers,weight servers, and a central server) that perform different functionsto track the positions of people and/or objects in a space. By spreadingthe functionality of the system amongst these various components, thesystem is capable of processing signals from more sensors (e.g., camerasand weight sensors). Due to the increase in the number of sensors, thesystem can track people and/or objects in a larger space. As a result,the system can be scaled to handle larger spaces (e.g., by addingadditional camera clients). Certain embodiments of the tracking systemare described below.

According to an embodiment, a system includes an array of cameras, afirst camera client, a second camera client, a camera server, aplurality of weight sensors, a weight server, and a central server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client receives a first plurality offrames of a first video from a first camera of the array of cameras.Each frame of the first plurality of frames shows the person within thespace. For a first frame of the first plurality of frames, the firstcamera client determines a first bounding area around the person shownin the first frame and generates a first timestamp of when the firstframe was received by the first camera client. For a second frame of thefirst plurality of frames, the first camera client determines a secondbounding area around the person shown in the second frame and generatesa second timestamp of when the second frame was received by the firstcamera client. The second camera client is separate from the firstcamera client. The second camera client receives a second plurality offrames of a second video from a second camera of the array of cameras.Each frame of the second plurality of frames shows the person within thespace. For a third frame of the second plurality of frames, the secondcamera client determines a third bounding area around the person shownin the third frame and generates a third timestamp of when the thirdframe was received by the second camera client. For a fourth frame ofthe second plurality of frames, the second camera client determines afourth bounding area around the person shown in the fourth frame andgenerates a fourth timestamp of when the fourth frame was received bythe second camera client.

The camera server is separate from the first and second camera clients.The camera server determines that the first timestamp falls within afirst time window and in response to determining that the firsttimestamp falls within the first time window, assigns coordinatesdefining the first bounding area to the first time window. The cameraserver also determines that the second timestamp falls within the firsttime window and in response to determining that the second timestampfalls within the first time window, assigns coordinates defining thesecond bounding area to the first time window. The camera server furtherdetermines that the third timestamp falls within the first time windowand in response to determining that the third timestamp falls within thefirst time window, assigns coordinates defining the third bounding areato the first time window. The camera server determines that the fourthtimestamp falls within a second time window that follows the first timewindow and in response to determining that the fourth timestamp fallswithin the second time window, assigns coordinates defining the fourthbounding area to the second time window.

The camera server also determines that coordinates assigned to the firsttime window should be processed and in response to determining thatcoordinates assigned to the first time window should be processed, thecamera server calculates, based at least on the coordinates defining thefirst bounding area and the coordinates defining the second boundingarea, a combined coordinate for the person during the first time windowfor the first video from the first camera and calculates, based at leaston the coordinates defining the third bounding area, a combinedcoordinate for the person during the first time window for the secondvideo from the second camera. The camera server also determines, basedat least on the combined coordinate for the person during the first timewindow for the first video from the first camera and the combinedcoordinate for the person during the first time window for the secondvideo from the second camera, a position of the person within the spaceduring the first time window.

The plurality of weight sensors are positioned within the space. Eachweight sensor of the plurality of weight sensors produces a signalindicative of a weight experienced by that weight sensor. The weightserver is separate from the first and second camera clients and thecamera server. The weight server determines, based at least on a signalproduced by a first weight sensor of the plurality of weight sensors,that an item positioned above the first weight sensor was removed. Thecentral server is separate from the first and second camera clients, thecamera server, and the weight server. The central server determines,based at least on the position of the person within the space during thefirst time window, that the person removed the item. Based at least onthe determination that the first person removed the item, the person ischarged for the item when the person exits the space.

According to another embodiment, a system includes an array of cameras,a first camera client, a second camera client, a camera server, aplurality of weight sensors, a weight server, and a central server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client, for each frame of a firstvideo received from a first camera of the array of cameras, determines abounding area around the person shown in that frame of the first videoand generates a timestamp of when that frame of the first video wasreceived by the first camera client. The second camera client, for eachframe of a second video received from a second camera of the array ofcameras, determines a bounding area around the person shown in thatframe of the second video and generates a timestamp of when that frameof the second video was received by the second camera client.

The camera server is separate from the first and second camera clients.The camera server, for each frame of the first video, assigns, based atleast on the timestamp of when that frame was received by the firstcamera client, coordinates defining the bounding area around the personshown in that frame to one of a plurality of time windows. For eachframe of the second plurality of frames, the camera server assigns,based at least on the timestamp of when that frame was received by thesecond camera client, coordinates defining the bounding area around theperson shown in that frame to one of the plurality of time windows. Fora first time window of the plurality of time windows, the camera servercalculates, based at least on the coordinates that (1) define boundingareas around the person shown in the first plurality of frames and (2)are assigned to the first time window, a combined coordinate for theperson during the first time window for the first video from the firstcamera and calculates, based at least on the coordinates that (1) definebounding areas around the person shown in the second plurality of framesand (2) are assigned to the first time window, a combined coordinate forthe person during the first time window for the second video from thesecond camera. The camera server determines, based at least on thecombined coordinate for the person during the first time window for thefirst video from the first camera and the combined coordinate for theperson during the first time window for the second video from the secondcamera, a position of the person within the space during the first timewindow.

The plurality of weight sensors are positioned within the space. Theweight server is separate from the first and second camera clients andthe camera server. The weight server determines, based at least on asignal produced by a first weight sensor of the plurality of weightsensors, that an item positioned above the first weight sensor wasremoved. The central server is separate from the first and second cameraclients, the camera server, and the weight server. The central serverdetermines, based at least on the position of the person within thespace during the first time window, that the person removed the item.

Certain embodiments of the tracking system perform an unconventionaltracking process that allows for some desynchronization amongst thecomponents of the system (e.g., camera clients and camera server).Generally, the system processes information according to time windows.These time windows may be set to be larger than the desynchronizationthat is expected to exist in the system. Information that is assigned toa time window is processed together. Thus, even if somedesynchronization exists amongst that information, it is neverthelessprocessed together within the same time window. In this manner, thetracking system can handle an increased amount of desynchronization,especially desynchronization that occurs as a result of the system beingscaled to include more components so that the system can handle a largerspace. As a result, the system can scale to handle larger spaces whilemaintaining reliability and accuracy. Certain embodiments of thetracking process are described below.

According to an embodiment, a system includes an array of cameras, afirst camera client, a second camera client, and a camera server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client receives a first plurality offrames of a first video from a first camera of the array of cameras.Each frame of the first plurality of frames shows the person within thespace. For a first frame of the first plurality of frames, the firstcamera client determines a first bounding area around the person shownin the first frame and generates a first timestamp of when the firstframe was received by the first camera client. For a second frame of thefirst plurality of frames, the first camera client determines a secondbounding area around the person shown in the second frame and generatesa second timestamp of when the second frame was received by the firstcamera client. For a third frame of the first plurality of frames, thefirst camera client determines a third bounding area around the personshown in the third frame and generates a third timestamp of when thethird frame was received by the first camera client.

The second camera client receives a second plurality of frames of asecond video from a second camera of the array of cameras. Each frame ofthe second plurality of frames shows the person within the space. For afourth frame of the second plurality of frames, the second camera clientdetermines a fourth bounding area around the person shown in the fourthframe and generates a fourth timestamp of when the fourth frame wasreceived by the second camera client. For a fifth frame of the secondplurality of frames, the second camera client determines a fifthbounding area around the person shown in the fifth frame and generates afifth timestamp of when the fifth frame was received by the secondcamera client.

The camera server is separate from the first and second camera clients.The camera server determines that the first timestamp falls within afirst time window and in response to determining that the firsttimestamp falls within the first time window, assigns coordinatesdefining the first bounding area to the first time window. The cameraserver also determines that the second timestamp falls within the firsttime window and in response to determining that the second timestampfalls within the first time window, assigns coordinates defining thesecond bounding area to the first time window. The camera server furtherdetermines that the third timestamp falls within a second time windowthat follows the first time window and in response to determining thatthe third timestamp falls within the second time window, assignscoordinates defining the third bounding area to the second time window.The camera server also determines that the fourth timestamp falls withinthe first time window and in response to determining that the fourthtimestamp falls within the first time window, assigns coordinatesdefining the fourth bounding area to the first time window. The cameraserver further determines that the fifth timestamp falls within thesecond time window and in response to determining that the fifthtimestamp falls within the second time window, assigns coordinatesdefining the fifth bounding area to the second time window.

The camera server also determines that coordinates assigned to the firsttime window should be processed and in response to determining thatcoordinates assigned to the first time window should be processed, thecamera server calculates, based at least on the coordinates defining thefirst bounding area and the coordinates defining the second boundingarea, a combined coordinate for the person during the first time windowfor the first video from the first camera and calculates, based at leaston the coordinates defining the fourth bounding area, a combinedcoordinate for the person during the first time window for the secondvideo from the second camera. After determining that coordinatesassigned to the first time window should be processed, the camera serverdetermines that coordinates assigned to the second time window should beprocessed and in response to determining that coordinates assigned tothe second time window should be processed, the camera serverscalculates, based at least on the coordinates defining the thirdbounding area, a combined coordinate for the person during the secondtime window for the first video from the first camera and calculates,based at least on the coordinates defining the fifth bounding area, acombined coordinate for the person during the second time window for thesecond video from the second camera.

According to another embodiment, a system includes an array of cameras,a first camera client, a second camera client, and a camera server. Thearray of cameras is positioned above a space. Each camera of the arrayof cameras captures a video of a portion of the space. The spacecontains a person. The first camera client receives a first plurality offrames of a first video from a first camera of the array of cameras.Each frame of the first plurality of frames shows the person within thespace. For each frame of the first plurality of frames, the first cameraclient determines a bounding area around the person shown in that frameand generates a timestamp of when that frame was received by the firstcamera client. The second camera client receives a second plurality offrames of a second video from a second camera of the array of cameras.Each frame of the second plurality of frames shows the person within thespace. For each frame of the second plurality of frames, the secondcamera client determines a bounding area around the person shown in thatframe and generates a timestamp of when that frame was received by thesecond camera client.

The camera server is separate from the first and second camera clients.The camera server, for each frame of the first plurality of frames,assigns, based at least on the timestamp of when that frame was receivedby the first camera client, coordinates defining the bounding areaaround the person shown in that frame to one of a plurality of timewindows and for each frame of the second plurality of frames, assigns,based at least on the timestamp of when that frame was received by thesecond camera client, coordinates defining the bounding area around theperson shown in that frame to one of the plurality of time windows.

The camera server also determines that coordinates assigned to a firsttime window of the plurality of time windows should be processed and inresponse to determining that coordinates assigned to the first timewindow should be processed, calculates, based at least on thecoordinates that (1) define bounding areas around the person shown inthe first plurality of frames and (2) are assigned to the first timewindow, a combined coordinate for the person during the first timewindow for the first video from the first camera and calculates, basedat least on the coordinates that (1) define bounding areas around theperson shown in the second plurality of frames and (2) are assigned tothe first time window, a combined coordinate for the person during thefirst time window for the second video from the second camera.

Certain embodiments include an unconventional arrangement of cameras andcamera clients that improve the resiliency of the camera system.Generally, the cameras are arranged in a rectangular grid that providescoverage for a physical space, and each camera is communicativelycoupled to one camera client. No camera is directly adjacent in the samerow or column of the grid to another camera that is communicativelycoupled to the same camera client. Cameras arranged along a diagonal ofthe grid are communicatively coupled to the same camera client. In thismanner, even if one camera client in the system were to go offline, thegrid still provides sufficient coverage for the physical space. As aresult, the arrangement of the cameras improves the resiliency of thesystem. Certain embodiments of the camera arrangement are describedbelow.

According to an embodiment, a system includes a first camera client, asecond camera client, a third camera client, and an array of cameras.The second camera client is separate from the first camera client. Thethird camera client is separate from the first and second cameraclients. The array of cameras is positioned above a space. The camerasin the array of cameras are arranged as a rectangular grid comprising afirst row, a second row, a third row, a first column, a second column,and a third column. The array includes first, second, third, fourth,fifth, and sixth cameras.

The first camera is positioned in the first row and the first column ofthe grid. The first camera is communicatively coupled to the firstcamera client. The first camera communicates a video of a first portionof the space to the first camera client. The second camera is positionedin the first row and the second column of the grid such that the secondcamera is directly adjacent to the first camera in the grid. The secondcamera is communicatively coupled to the second camera client. Thesecond camera communicates a video of a second portion of the space tothe second camera client. The third camera is positioned in the firstrow and the third column of the grid such that the third camera isdirectly adjacent to the second camera in the grid. The third camera iscommunicatively coupled to the third camera client. The third cameracommunicates a video of a third portion of the space to the third cameraclient. The fourth camera is positioned in the second row and the firstcolumn of the grid such that the fourth camera is directly adjacent tothe first camera in the grid. The fourth camera is communicativelycoupled to the second camera client. The fourth camera communicates avideo of a fourth portion of the space to the second camera client. Thefifth camera is positioned in the second row and the second column ofthe grid such that the fifth camera is directly adjacent to the fourthcamera and the second camera in the grid. The fifth camera iscommunicatively coupled to the third camera client. The fifth cameracommunicates a video of a fifth portion of the space to the third cameraclient. The sixth camera is positioned in the third row and the firstcolumn of the grid such that the sixth camera is directly adjacent tothe fourth camera in the grid. The sixth camera is communicativelycoupled to the third camera client. The sixth camera communicates avideo of a sixth portion of the space to the third camera client.

According to another embodiment, a system includes a plurality of cameraclients and an array of cameras. The plurality of camera clientsincludes a number of camera clients. The array of cameras is positionedabove a space. Each camera in the array of cameras communicates a videoof a portion of the space to only one camera client of the plurality ofcamera clients. The cameras in the array of cameras are arranged suchthat each camera client of the plurality of camera clients iscommunicatively coupled to at least one camera in an N×N portion of thearray. N is the number of camera clients in the plurality of cameraclients minus one.

Certain embodiments include an unconventional rack for holding items.The rack includes a base and panels for holding shelves and weightsensors. The weight sensors are wired to a circuit board located in adrawer in the base. The wires run from the weight sensors throughcavities and spaces defined by the shelves, panels, and base. Certainembodiments of the rack are described below.

According to an embodiment, a system includes a circuit board and arack. The rack includes a base, a panel, a shelf, a first weight sensor,a second weight sensor, a first wire, and a second wire. The baseincludes a bottom surface, a first side surface, a second side surface,a third side surface, a top surface, and a drawer. The first sidesurface is coupled to the bottom surface of the base. The first sidesurface of the base extends upwards from the bottom surface of the base.The second side surface is coupled to the bottom and first side surfacesof the base. The second side surface of the base extends upwards fromthe bottom surface of the base. The third side surface is coupled to thebottom and second side surfaces of the base. The third side surface ofthe base extends upwards from the bottom surface of the base. The topsurface is coupled to the first, second, and third side surfaces of thebase such that the bottom and top surfaces of the base and the first,second, and third side surfaces of the base define a space. The topsurface of the base defines a first opening into the space. The draweris positioned within the space. The circuit board is positioned withinthe drawer.

The panel is coupled to the base and extends upwards from the base. Thepanel defines a second opening that extends along a width of the panel.The shelf is coupled to the panel such that the shelf is positionedvertically higher than the base and such that the shelf extends awayfrom the panel. The shelf includes a bottom surface, a front surfacethat extends upwards from the bottom surface of the shelf, and a backsurface that extends upwards from the bottom surface of the shelf. Theback surface of the shelf is coupled to the panel. The back surface ofthe shelf defines a third opening. A portion of the third opening alignswith a portion of the second opening.

The first weight sensor is coupled to the bottom surface of the shelfand positioned between the front surface of the shelf and the backsurface of the shelf The second weight sensor is coupled to the bottomsurface of the shelf and positioned between the front surface of theshelf and the back surface of the shelf. The first wire is coupled tothe first weight sensor and the circuit board The first wire extendsfrom the first weight sensor through the second and third openings anddownwards into the space through the first opening. The second wire iscoupled to the second weight sensor and the circuit board. The secondwire extends from the second weight sensor through the second and thirdopenings and downwards into the space through the first opening.

Certain embodiments may include none, some, or all of the abovetechnical advantages discussed above. One or more other technicaladvantages may be readily apparent to one skilled in the art from thefigures, descriptions, and claims included herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following description, taken in conjunction with theaccompanying drawings, in which:

FIG. 1A-1C illustrates an example store that defines a physical space;

FIG. 2 illustrates a block diagram of an example tracking system for usein the physical store;

FIGS. 3A-3T illustrate an example camera subsystem and its operation inthe tracking system;

FIGS. 4A-4D illustrate an example light detection and ranging subsystemand its operation in the tracking system;

FIGS. 5A-5J illustrate an example weight subsystem and its operation inthe tracking system;

FIGS. 6A-6C illustrate the operation of an example central server foruse in conjunction with the tracking system; and

FIG. 7 illustrates an example computer.

DETAILED DESCRIPTION

Embodiments of the present disclosure and its advantages are bestunderstood by referring to FIGS. 1A through 7 of the drawings, likenumerals being used for like and corresponding parts of the variousdrawings. Additional information is disclosed in U.S. Patent ApplicationNo. ______ entitled, “Customer-Based Video Feed” (attorney docket no.090278.0187) and U.S. Patent Application No. ______ entitled, “TopviewObject Tracking Using a Sensor Array” (attorney docket no. 090278.0180)which are both hereby incorporated by reference herein as if reproducedin their entirety.

Position tracking systems are used to track the physical positions ofpeople and/or objects in a physical space (e.g., a store). These systemstypically use a sensor (e.g., a camera) to detect the presence of aperson and/or object and a computer to determine the physical positionof the person and/or object based on signals from the sensor. In a storesetting, other types of sensors can be installed to track the movementof inventory within the store. For example, weight sensors can beinstalled on racks and shelves to determine when items have been removedfrom those racks and shelves. By tracking both the positions of personsin a store and when items have been removed from shelves, it is possiblefor the computer to determine which user in the store removed the itemand to charge that user for the item without needing to ring up the itemat a register. In other words, the person can walk into the store, takeitems, and leave the store without stopping for the conventionalcheckout process.

For larger physical spaces (e.g., convenience stores and grocerystores), additional sensors can be installed throughout the space totrack the position of people and/or objects as they move about thespace. For example, additional cameras can be added to track positionsin the larger space and additional weight sensors can be added to trackadditional items and shelves. There is a limit, however, to the numberof sensors that can be added before the computing capabilities of thecomputer are reached. As a result, the computing power of the computerlimits the coverage area of the tracking system.

One way to scale these systems to handle larger spaces is to addadditional computers and to divide the sensors amongst these computerssuch that each computer processes signals from a subset of the sensors.However, dividing the sensors amongst multiple computers introducessynchronization issues. For example, the sensors may not communicatesignals to their respective computers at the same time orsimultaneously. As another example, sensors may have different latencieswith their respective computers and thus, it may take more time forsignals from one sensor to reach a computer than signals from anothersensor. As a result, the sensors and computers become desynchronizedwith each other and it becomes more difficult for the computers todetermine, in a cohesive way, the position of persons or objects in thespace and when items were removed.

This disclosure contemplates an unconventional, distributed trackingsystem that can scale to handle larger spaces. The system uses an arrayof cameras, multiple camera clients, a camera server, weight sensors, aweight server, and a central server to determine which person in thespace took an item and should be charged for the item. The cameraclients each process frames of videos from a different subset of camerasof the array of cameras. Each camera client determines coordinates forpeople detected in the frames and then timestamps these coordinatesbased on when the frames were received by the camera client. The cameraclients then communicate the coordinates and timestamps to a cameraserver that is responsible for coordinating the information from thecamera clients. The camera server determines, based on the coordinatesand timestamps from the camera clients, the positions of people in thespace. The weight server processes signals from the weight sensors todetermine when items were removed from shelves in the space. The centralserver uses the positions of people in the space from the camera serverand the determinations from the weight server of when items were removedfrom shelves to determine which people in the space took which items andshould therefore be charged. The system will be described in more detailusing FIGS. 1A-7.

Generally, the camera server protects against desynchronization byassigning the coordinates from the multiple camera clients to windows oftime based on the timestamps. The camera server then processes thecoordinates assigned to a particular time window to determine overallcoordinates for people in the space during that time window. Theduration of the time window can be set to be larger than thedesynchronization that is expected to occur to mitigate the effects ofdesynchronization. For example, if the cameras and camera clients areexpected to desynchronize by a few milliseconds, then the time windowcan be set to last 100 milliseconds to counteract the desynchronization.In this manner, the number of cameras and camera clients can beincreased to scale the system to handle any suitable space. The cameras,camera clients, and camera server will be described in more detail usingFIGS. 1A-3Q.

This disclosure also contemplates an unconventional way of wiringcameras in the array of cameras to the camera clients. The cameras arearranged as a rectangular grid above the space. Each camera in the gridis wired to a particular camera client according to certain rules. Forexample, no two cameras that are directly adjacent to one another in thesame row or column of the grid are wired to the same camera client. Asanother example, cameras that are arranged along a diagonal in the gridare wired to the same camera client. In this manner, a small area of thegrid should include cameras that are wired to each and every cameraclient in the system. As a result, even if one camera client were to gooffline (e.g., maintenance, error, or crash), there would still beenough coverage from the remaining camera clients to track the positionsof people in the area. Thus, this arrangement of the cameras improvesthe resiliency of the system. The camera array will be described in moredetail using FIGS. 3A-3E.

This disclosure further contemplates an unconventional rack and shelfdesign that integrates weight sensors for tracking when items have beenremoved from the racks and shelves. Generally, the rack includes a base,a vertical panel, and a shelf. The base forms an enclosed space in whicha printed circuit board is positioned, and the base includes a drawerthat opens to provide access to the enclosed space and the circuitboard. The vertical panel is attached to the base, and the shelf isattached to the vertical panel. Weight sensors are positioned within theshelf. The base, panel, and shelf each define a cavity. The cavity inthe shelf and the cavity in the panel are at least partially aligned.Each weight sensor communicates signals to the printed circuit boardthrough a wire that runs from that weight sensor, through the cavity ofthe shelf, the cavity of the panel, and the cavity of the base, to thecircuit board. The rack and shelf design will be described in moredetail using FIGS. 5A-5K.

The system may also include a light detection and ranging (LiDAR)subsystem that will be described in more detail using FIGS. 4A-4D. Thesystem also includes a central server that ties together the camerasubsystem, weight subsystem, and LiDAR subsystem. The central serverwill be described in more detail using FIGS. 6A-6C.

I. System Overview

FIGS. 1A-1D show the tracking system installed in an example storespace. As discussed above, the tracking system may be installed in astore space so that shoppers need not engage in the conventionalcheckout process. Although the example of a store space is used in thisdisclosure, this disclosure contemplates that the tracking system may beinstalled and used in any type of physical space (e.g., a warehouse, astorage center, an amusement park, an airport, an office building,etc.). Generally, the tracking system (or components thereof) is used totrack the positions of people and/or objects within these spaces for anysuitable purpose. For example, at an airport, the tracking system cantrack the positions of travelers and employees for security purposes. Asanother example, at an amusement park, the tracking system can track thepositions of park guests to gauge the popularity of attractions. As yetanother example, at an office building, the tracking system can trackthe positions of employees and staff to monitor their productivitylevels.

FIG. 1A shows an example store 100. Store 100 is a physical space inwhich shoppers can purchase items for sale. As seen in FIG. 1A, store100 is a physical building that includes an entryway through whichshoppers can enter and exit store 100. A tracking system may beinstalled in store 100 so that shoppers need not engage in theconventional checkout process to purchase items from store 100. Thisdisclosure contemplates that store 100 may be any suitable physicalspace. For example, store 100 may be a convenience store or a grocerystore. This disclosure also contemplates that store 100 may not be aphysical building, but a physical space or environment in which shoppersmay shop. For example, store 100 may be a grab and go pantry at anairport, a kiosk in an office building, an outdoor market at a park,etc.

FIG. 1B shows portions of the interior of store 100. As seen in FIG. 1B,store 100 contains shoppers 105, racks 115, and gates 125. Shoppers 105may have entered store 100 through one of gates 125, which allow entryand exit from store 100. Gates 125 prevent shoppers 105 from enteringand/or exiting the store unless gates 125 are opened.

Gates 125 may include scanners 110 and 120. Scanners 110 and 120 mayinclude a QR code scanner, a barcode scanner, or any other suitable typeof scanner that can receive an electronic code embedded withinformation, such as information that uniquely identifies a shopper 105.Shoppers 105 may scan a personal device (e.g., a smartphone) on scanners110 to enter store 100. When a shopper 105 scans a personal device onscanners 110, the personal device may provide scanners 110 an electroniccode that uniquely identifies the shopper 105. When the shopper 105 isidentified and/or authenticated, gate 125 that includes scanner 110opens to allow the shopper 105 into store 100. Each shopper 105 may haveregistered an account with store 100 to receive an identification codefor the personal device.

After entering store 100, shoppers 105 may move around the interior ofstore 100. As the shoppers 105 move throughout the space, shoppers 105may shop for items 130 by removing items 130 from racks 115. As seen inFIG. 1B, store 100 includes racks 115 that hold items 130. When shopper105 wishes to purchase a particular item 130, shopper 105 can removethat item 130 from rack 115. Shoppers 105 can remove multiple items 130from store 100 to purchase those items 130.

When shopper 105 has completed shopping for items 130, shopper 105approaches gates 125. In certain embodiments, gates 125 willautomatically open so that shopper 105 can leave store 100. In otherembodiments, shopper 105 scans a personal device on scanners 120 beforegates 125 will open to allow shopper 105 to exit store 100. When shopper105 scans a personal device on scanner 120, the personal device mayprovide an electronic code that uniquely identifies shopper 105 toindicate that shopper 105 is leaving store 100. When shopper 105 leavesstore 100, an account of shopper 105 is charged for the items 130 thatshopper 105 removed from store 100.

FIG. 1C shows the interior of store 100 along with a tracking system 132that allows shopper 105 to simply leave store 100 with items 130 withoutengaging in a conventional checkout process. As seen in FIG. 1C, thetracking system 132 includes an array of sensors 135 positioned on theceiling of store 100. The array of sensors 135 may provide coverage forthe interior space of store 100. Sensors 135 are arranged in a gridpattern across the ceiling of store 100, as explained in further detailwith respect to FIGS. 3A-3E. Sensors 135 may be used to track thepositions of shoppers 105 within the space of store 100. This disclosurecontemplates sensors 135 being any suitable sensors (e.g., cameras,light detection and range sensors, millimeter wave sensors, etc.).

The tracking system 132 also includes weight sensors 140 positioned onthe racks 115. Weight sensors 140 can detect the weight of items 130positioned on racks 115. When an item 130 is removed from the rack 115,the weight sensor 140 may detect a decrease in weight. The trackingsystem 132 may use that information to determine that a particular item130 was removed from the rack 115.

The tracking system 132 includes a computer system 145. Computer system145 may include multiple computers that operate together to determinewhich shopper 105 took which items 130 from racks 115. The components ofcomputer system 145 and their operation will be described in more detailusing FIGS. 2 through 7. Generally, computer system 145 uses informationfrom sensors 135 and weight sensors 140 to determine which shopper 105removed which items 130 from store 100. In this manner, the shopper 105may be automatically charged for items 130 when the shopper 105 leavesstore 100 through gates 125.

FIG. 2 illustrates a block diagram of an example tracking system 132 foruse in store 100. As seen in FIG. 2, the tracking system 132 includes acamera subsystem 202, a light detection and range (LiDAR) subsystem 204,and a weight subsystem 206. The tracking system 132 includes varioussensors 135, such as cameras 205, light detection and range (LiDAR)sensors 210, and weight sensors 215. These sensors 135 arecommunicatively coupled to various computers of a computer system 145.For example, the camera subsystem 202 includes cameras 205 that arecommunicatively coupled to one or more camera clients 220. These cameraclients 220 are communicatively coupled to a camera server 225. TheLiDAR subsystem 204 includes LiDAR sensors 210 that are communicativelycoupled to a LiDAR server 230. The weight subsystem 206 includes weightsensors 215 that are communicatively coupled to a weight server 235. Thecamera server 225, LiDAR server 230, and weight server 235 arecommunicatively coupled to a central server 240.

Generally, cameras 205 produce videos of portions of the interior of aspace. These videos may include frames or images of shoppers 105 withinthe space. The camera clients 220 process the frames from the cameras205 to detect shoppers 105 within the frames and to assign framecoordinates to those shoppers 105. The camera server 225 generallyprocesses frame data from the camera clients 220 to determine thephysical position of shoppers 105 within the space. LiDAR sensors 210generally produce coordinates of shoppers 105 within a space. LiDARserver 230 processes these coordinates to determine the position ofshoppers 105 within a space. Weight sensors 215 detect the weight ofitems 130 on racks 115 within the space. Weight server 235 processesthese weights to determine when certain items 130 have been removed fromthe racks 115.

Central server 240 processes position information for shoppers 105 fromcamera server 225 and LiDAR server 230 and weight information fromweight server 235 to determine which shopper 105 removed which items 130from the racks 115. These shoppers 105 may then be charged for thoseitems 130 when the shoppers 105 leave the space. The operation of thesecomponents will be described in more detail using FIGS. 3A through 6C.

In one embodiment, each of the components of tracking system 132 (e.g.camera clients 220, camera server 225, LiDAR server 230, weight server235, and central server 240) is a distinct computing device separatefrom the other components of tracking system 132. For example, each ofthese components may include its own processor, memory, and physicalhousing. In this manner, the components of tracking system 132 aredistributed to provide additional computing power relative to a trackingsystem that includes only one computer.

II. Camera Subsystem

FIGS. 3A-3R show an example camera subsystem 202 and its operation inthe tracking system 132. As discussed above, the camera subsystem 202includes cameras 205, camera clients 220, and a camera server 225.Generally, the cameras 205 capture video of a space and send the videosto the camera clients 220 for processing. These videos are a sequence offrames or images of the space. The camera clients 220 detect thepresence of people (e.g., shoppers 105) in the frames and determinecoordinates in the frames (may also be referred to as “framecoordinates”) for those people. The camera server 225 analyzes the framecoordinates from each camera client 220 to determine physical positionsof the people in the space.

1. Camera Array

FIG. 3A illustrates an example camera array 300. As shown in FIG. 3A,camera array 300 includes multiple cameras 305. Although this disclosureshows camera array 300 including twelve cameras 305, camera array 300may include any suitable number of cameras 305. Generally, camera array300 is positioned above a space so that cameras 305 can capture overheadvideos of portions of the space. These videos may then be processed byother components of the camera subsystem 202 to determine the physicalposition of people (e.g., shoppers 105) within the space. In the exampleof FIG. 3A, camera array 300 includes cameras 305A, 305B, 305C, 305D,305E, 305F, 305G, 305H, 3051, 305J, 305K, and 305L.

Generally, cameras 305 in camera array 300 are arranged to form arectangular array. In the example of FIG. 3A, camera array 300 is a 3×4array of cameras 305 (e.g., three rows and four columns of cameras 305).Camera array 300 may include any suitable number of cameras 305 arrangedin an array of any suitable dimensions.

Each camera 305 of camera array 300 is communicatively coupled to acamera client 220. In the example of FIG. 3A, each camera 305 of cameraarray 300 is communicatively coupled to one of camera client 1 220A,camera client 2 220B, or camera client 3 220C. Each camera 305communicates captured video to the camera client 220 to which the camera305 is communicatively coupled. The cameras 305 are communicativelycoupled to the camera clients 220 according to particular rules toimprove the resiliency of the tracking system 132. Generally, thecameras 305 are communicatively coupled to the camera clients 220 sothat even if one camera client 220 goes offline, the coverage of aphysical space provided by the cameras 305 communicatively coupled tothe remaining camera clients 220 is sufficient to allow the trackingsystem 132 to continue tracking the position of people within the space.

Cameras 305 are communicatively to camera clients 220 using any suitablemedium. For example, cameras 305 may be hardwired to camera clients 220.As another example, cameras 305 may wirelessly couple to camera clients220 using any suitable wireless protocol (e.g., WiFi). Cameras 305communicate captured videos through the communication medium to thecamera clients 220.

Cameras 305 may be any suitable devices for capturing videos of thespace. For example, cameras 305 may be three-dimensional cameras thatcan capture two-dimensional video of the space (e.g., x-y plane) andalso detect the heights of people and/or objects in the video (e.g., zplane). As another example, cameras 305 may be two-dimensional camerasthat capture two-dimensional videos of the space. Camera array 300 mayinclude a mixture of different types of cameras 305.

FIG. 3B illustrates the coverage provided by cameras 305 of a cameraarray 300. As seen in FIG. 3B, a floor space is covered by differentfields of view 310. Each field of view 310 is provided by a camera 305of camera array 300. For example, field of view 310A is provided bycamera 305A. Field of view 310B is provided by camera 305B. Field ofview 310C is provided by camera 305C, and so forth. Each field of view310 is generally rectangular in shape and covers a portion of the floorspace. Each camera 305 captures video of the portion of the floor spacethat is covered by that camera's 305 field of view 310. For example,camera 305A captures video of the portion of the floor space covered byfield of view 310A. Camera 305B captures video of the portion of thefloor space covered by field of view 310B. Camera 305C captures video ofthe portion of the floor space covered by field of 310C, and so forth.

Each field of view 310 is shaded differently than its neighbors todistinguish the fields of view 310. Fields of view 310A, 310C, 3101, and310K are shaded using lines that slant downwards to the right. Fields ofview 310B, 310D, 310J, and 310L are shaded using lines that slantupwards to the right. Fields of view 310E and 310G are shaded usinghorizontal lines, and fields of view 310F and 310H are shaded usingvertical lines. The shading of each field of view 310 is meant todistinguish that field of view 310 from other, directly adjacent fieldsof view 310. The shading is not meant to indicate a particularcharacteristic of the field of view 310. In other words, even thoughcertain fields of view 310 share the same shading, the similar shadingdoes not indicate that these fields of view 310 share certaincharacteristics (e.g., size, coverage, duration, and/or shape). Fieldsof view 310 may share one or more of these characteristics irrespectiveof their individual shading.

As seen in FIG. 3B, each field of view 310 overlaps with other fields ofview 310. For example, field of view 310A overlaps fields of view 310B,310E, and 310F. As another example, field of view 310F overlaps withfields of view 310A, 310B, 310C, 310E, 310G, 3101, 310J, and 310K. Likefields of view 310A and 310F, other fields of view 310 (e.g., fields ofview 310B, 310C, 310D, 310E, 310G, 310H, 3101, 310J, 310K, and 310L)also overlap neighboring fields of view 310. The shading in theoverlapping regions is a combination of the shadings in the individualfields of view that form the overlapping regions. For example, theoverlapping region formed by fields of view 310A and 310B includesslanted lines running in opposite directions. As another example, theoverlapping region formed by fields of view 310A, 310B, 310E, and 310Fincludes slanted lines running in opposite directions, horizontal lines,and vertical lines.

The overlapping fields of view 310 may be a result of the proximity ofcameras 305 to each other in camera array 300. Generally, by overlappingfields of view 310, certain portions of the floor space can be capturedby multiple cameras 305 of the camera array 300. As a result, even ifcertain cameras 305 go offline, there may still be sufficient coverageprovided by the remaining cameras 305 for the tracking system 132 tooperate. Additionally, the overlapping fields of view 310 may improvetracking the positions of people (e.g., shoppers 105) as they move aboutthe space.

FIG. 3C illustrates an example camera grid 315. As seen in FIG. 3C,camera grid 315 includes a number of rows and a number of columnscorresponding to the number of rows and columns in camera array 300.Each box of camera grid 315 represents a camera 305 of camera array 300.Camera grid 315 shows how the cameras 305 of camera array 300 arecommunicatively coupled to camera clients 220. Using the previousexample of FIG. 3A, camera grid 315 shows that cameras 305A, 305D, 305G,and 305J are communicatively coupled to camera client 1 220A. Cameragrid 315 also shows that cameras 305B, 305E, 305H, and 305K arecommunicatively coupled to camera client 2 220B. Camera grid 315 furthershows that cameras 305C, 305F, 3051, and 305L are communicativelycoupled to camera client 3 220C.

Camera grid 315 shows that cameras 305 are communicatively coupled tocamera clients 220 according to particular rules. For example, a camera305 that is communicatively coupled to a particular camera client 220 isnot directly adjacent in the same row or the same column of camera grid315 to another camera 305 that is communicatively coupled to the samecamera client 220. As seen in FIG. 3C, for example, camera 305A isdirectly adjacent in the same row or the same column of camera grid 315to cameras 305B and 305E. Camera 305A is communicatively coupled tocamera client 1 220A while cameras 305B and 305E are communicativelycoupled to camera client 2 220B. Camera 305F is directly adjacent in thesame row or the same column of camera grid 315 to cameras 305B, 305E,305G, and 305J. Camera 305F is communicatively to camera client 3 220C,while cameras 305B, 305E, 305G, and 305J are communicatively coupled tocamera client 1 220A or camera client 2 220B.

As another example, a camera 305 that is communicatively coupled to aparticular camera client 220 is diagonal in camera grid 315 to anothercamera 305 that is communicatively coupled to the same camera client220. As seen in FIG. 3C, for example, cameras 305D, 305G, and 305J arediagonal to each other and are communicatively coupled to camera client1 220A. Cameras 305C, 305F, and 3051 are diagonal to each other and areall communicatively coupled to camera client 3 220C.

A consequence of arranging cameras 305 in this manner is that eachcamera client 220 is communicatively coupled to at least one camera 305in a portion of camera grid 315. As seen in the example of FIG. 3C, eachof camera client 1 220A, camera client 2 220B, and camera client 3 220Cis communicatively coupled to at least one camera in any 2×2 portion ofcamera grid 315. As a result, even if one camera client 220 were to gooffline, the other cameras in the 2×2 portion can still providesufficient coverage of that 2×2 portion to allow the tracking system 132to operate. Thus, the resiliency of the tracking system 132 is improved.

Although the previous example used a certain number of cameras 305 and acertain number of camera clients 220, the tracking system 132 may useany suitable number of cameras 305 and any suitable number of cameraclients 220 to provide a desired level of overlap, scalability, andresiliency. FIG. 3D shows an example camera array 300 that includesadditional cameras 305. The example of FIG. 3D also includes additionalcamera clients 220: camera client 1 220A through camera client N 220D.The cameras 305 in camera array 300 may be communicatively coupled tocamera clients 220 according to the same rules or principles describedin FIGS. 3A through 3C.

FIG. 3E shows how the cameras 305 may communicatively couple to thecamera clients 220. As seen in FIG. 3E, camera grid 315 includes anumber of rows and a number of columns. Across a row, the cameras 305are communicatively coupled to the camera clients 220 in a sequentialfashion. After a camera 305 is communicatively coupled to camera clientN 220 d, the sequence repeats until the end of the row is reached.Similarly, the cameras 305 in a column are sequentially coupled tocamera clients 220. After a camera 305 is communicatively coupled tocamera client N 220 d, the pattern repeats.

As shown in FIGS. 3D and 3E, the tracking system 132 may be scaled toinclude any number of cameras 305 and any number of camera clients 220.Generally, a camera 305 that is communicatively coupled to a particularcamera client 220 is not directly adjacent in the same row or the samecolumn of camera grid 315 to another camera 305 that is communicativelycoupled to the same camera client 220.

Additionally, cameras 305 along a diagonal of camera grid 315 arecommunicatively coupled to the same camera client 220. Furthermore, eachcamera client 220 is communicatively coupled to at least one camera 305in a portion of camera grid 315. The dimensions of the portion maydepend upon the number of camera clients 220 in the tracking system 132.Generally, the dimensions of the portion are one less than the number ofcamera clients 220 in the tracking system 132. So, in the examples ofFIGS. 3D and 3E, the dimensions of the portion are (N−1)×(N−1).

2. Initialization

FIG. 3F shows the initialization of the camera subsystem 202. As seen inFIG. 3F, the camera subsystem 202 includes a camera array 300, cameraclient 1 220A, camera client 2 220B, camera client 3 220C, and cameraserver 225. Camera subsystem 202, may include any suitable number ofcamera arrays 300, camera clients 220, and camera servers 225.Generally, during initialization, the cameras 305 of the camera array300 start up and begin sending videos 302 to camera clients 220.Additionally, camera clients 220 and camera server 225 synchronizeinternal clocks 304. After the cameras 305 in camera array 300 havestarted up and after the internal clocks 304 are synchronized, cameraclients 220 may begin processing videos 302 and communicatinginformation to camera server 225 to perform the tracking operations ofthe camera subsystem 202.

During initialization, the cameras 305 of camera array 300 may power onand perform a startup sequence. For example, the components of thecameras 305 may boot up and/or warm-up. The cameras 305 may then begincapturing video footage and communicating videos 302 to their respectivecamera clients 220. The cameras 305 of camera array 300 may takedifferent amounts of time to initialize. For example, certain cameras305 may take a shorter or longer amount of time to initialize than othercameras 305 of camera array 300. Because the cameras 305 of camera array300 do not wait for the other cameras 305 of camera array 300 tocomplete initialization before sending videos 302 to camera clients 220,the cameras 305 of camera array 300 may each begin sending videos 302 tocamera clients 220 at different times. As a result, videos 302, and inparticular, the frames of videos 302, may be desynchronized from theframes of other videos 302. In other words, the frames of these videos302 are not being captured and sent by their respective cameras 305simultaneously or at the same time. Consequentially, the frames of thesevideos 302 do not arrive at the camera clients 220 simultaneously or atthe same time.

During initialization, camera clients 220 and camera server 225 power onand/or perform a bootup sequence. After booting up, camera clients 220and camera server 225 synchronize their internal clocks 304. In theexample of FIG. 3F, camera client 1 220A has an internal clock 1 304A.Camera client 2 220B has an internal clock 2 304B. Camera client 3 220Chas an internal clock 3 304C. Camera server 225 has an internal clock 4304D. Camera clients 220 and camera server 225 may synchronize theirinternal clocks 304 in any suitable manner. For example, camera clients220 and camera server 225 may synchronize their internal clocks 304using a synchronization protocol, such as the Network Time Protocol(NTP) or the Precision Time Protocol (PTP). Although a synchronizationprotocol may be used to synchronize the internal clocks 304 of cameraclients 220 and camera server 225, this does not mean that theseinternal clocks 304 show exactly the same time or are perfectlysynchronized with each other. As a result, there may still be a level ofdesynchronization amongst camera clients 220 and camera server 225.

Camera clients 220 may track the cameras 305 of camera array 300 thathave completed initialization by tracking which cameras 305 havecommunicated videos 302 to camera clients 220. When camera clients 220determine that each camera 305 of camera array 300 have begun sendingvideos 302 to camera clients 220, camera clients 220 may determine thatcamera array 300 has finished initialization. In response to thatdetermination, camera clients 220 may begin processing the frames of thevideos 302 and communicating information from those frames to cameraserver 225. Camera server 225 may then analyze the information fromcamera clients 220 to determine the physical position of people and/orobjects within a space.

3. Camera Clients

FIGS. 3G-3I show the operation of camera clients 220 in the camerasubsystem 202. Generally, camera clients 320 process videos 302 fromcameras 305. Camera clients 320 may identify people or objects withinthe frames 320 of these videos 302 and determine coordinates 322 forthese people or objects. Camera clients 320 may also generate timestamps324 (e.g., by using internal clocks 304) that indicate when the cameraclients 320 received particular frames 320. Camera clients 320communicate these timestamps 324 and coordinates 322 to camera server225 for further processing.

FIGS. 3G-3I show the operation of camera clients 210 as an event in astore 100 unfolds. During this event, for example, a first shopper 105(e.g., a man) removes an item 130 from a shelf in the store 100 and asecond shopper 105 (e.g., a woman) moves towards the shelf. Cameraclients 320 analyze frames 320 of videos 302 to determine coordinates322 for the man and the woman in the frames 320.

As seen in FIG. 3G, a man is standing near a shelf and a woman isstanding further away from the shelf. Two cameras 305A and 305B arepositioned above the space and capture video 302 of the man and thewoman and the shelf These cameras 305A and 305B send their videos 302 totwo different camera clients 220A and 220B. Camera 305A sends video 305to camera client 220A. Camera 305B sends video 305 to camera client220B.

Camera client 220A receives video 305 from camera 305A, and specificallya frame 320A of that video 305. Camera client 220A processes the frame320A. As seen in frame 320A, the man is standing near the shelf and thewoman is standing further away from the shelf. Camera client 220Aprocesses frame 320A to determine bounding areas 325A and 325B aroundthe man and the woman. In the example of FIG. 3G, bounding areas 325Aand 325B are rectangular areas that surround the man and the woman,respectively. Bounding areas 325A and 325B approximate the positions ofthe man and the woman in the frame. This disclosure contemplates cameraclients 220 determining bounding areas 325 that are of any suitableshape and of any suitable size. For example, bounding areas 325 may becircular or may be irregularly shaped (e.g, so as to follow the contoursof the shopper 105 in the frames 320).

Camera client 220A determines coordinates 322 that define the boundingareas 325A and 325B within frames 320A and 320B (also referred to as“frame coordinates”). In the example of FIG. 3G, camera client 228determines coordinates 322 (x₁, y₁) and (x₂, y₂) for bounding area 325Aand coordinates 322 (x₃, y₃) and (x₄, y₄) for bounding area 325B. Thesecoordinates 322 do not represent absolute coordinates in the physicalspace, but rather coordinates within the frame 320A. Camera clients 220may determine any suitable number of coordinates 322 for bounding areas325.

Camera client 220A then generates frame data 330A that containsinformation about frame 320A. As seen in FIG. 3G, frame data 330Aincludes an identifier for camera 305A (e.g., “camera=1”). Camera client220A may also generate a timestamp 324 (e.g., using internal clock 304)that indicates when frame 320A was received by camera client 220A. Inthe example of FIG. 3G, that timestamp 324 is t₁. Frame data 320A alsoincludes information about the people or objects within frame 320A. Inthe example of FIG. 3G, frame data 330A includes information for anobject 1 and an object 2. Object 1 corresponds to the man and object 2corresponds to the woman. Frame data 330A indicates the coordinates 322for the man (x₁, y₁) and (x₂, y₂) along with a height of the man z₁. Asdiscussed previously, cameras 305 may be three-dimensional cameras thatcan detect the height of objects and/or people. Cameras 305 may haveprovided the heights of the man and the woman to the camera clients 320.In the example of FIG. 3G, camera 305A may have detected the heights ofthe man and the woman to be z₁ and z₂, respectively. Frame data 330Aalso includes information for the woman including the coordinates 322(x₃, y₃) and (x₄, y₄) and the height z₂. Camera client 220A maycommunicate frame data 330A to camera server 225 when frame data 330A isready.

In a corresponding manner, camera client 220B may process video 302 fromcamera 305B. As seen in FIG. 3G, camera client 220B receives a frame320B from camera 305B. Because camera 305B is at a different positionthan camera 305A, frame 320B will show a slightly different perspectiveof the event in the store 100 than frame 320A. Camera client 220Bdetermines bounding areas 325C and 325D around the man and the woman,respectively. Camera client 220B determines frame coordinates 322 (x₁,y₁) and (x₂, y₂) for bounding area 325C, and frame coordinates 322 (x₃,y₃) and (x₄, y₄) for bounding area 325D. Camera client 220B alsodetermines and generates a timestamp 324 t₂ (e.g., using internal clock304) that indicates when camera client 220B received frame 320B. Cameraclient 220B then generates frame data 330B for frame 320B. Frame data330B indicates that frame 320B was generated by camera 305B and wasreceived by camera client 220B at t₂. Frame data 330B also indicatesthat a man and a woman were detected in frame 320B. The man correspondsto coordinates 322 (x₁, y₁) and (x₂, y₂) and has a height z₁. The womancorresponds to coordinates 322 (x₃, y₃) and (x₄, y₄) and has a heightz₂. Camera client 220B communicates frame data 320B to camera server 225when frame data 320B is ready.

The coordinates 322 generated by camera clients 220A and 220B for framedata 330A and 330B may be the coordinates within a particular frame 320and not the coordinates within the physical space. Additionally,although the same subscripts have been used for the coordinates 322 inframe data 330A and 330B, this does not mean that these coordinates 322are the same. Rather, because cameras 305A and 305B are in differentpositions, it is likely that the coordinates 322 in frame 330A aredifferent from the coordinates 322 in frame data 330B. Camera clients220A and 220B are determining the coordinates 322 of the bounding areas325 within the frames 320 and not within the physical space. Cameraclients 220A and 220B determine these local coordinates 322independently of each other. The subscripts indicate a sequence ofcoordinates 322 generated by the individual camera clients 220. Forexample (x₁, y₁) indicates the first coordinate 322 generated by cameraclient 220A and the first coordinate 322 generated by camera client220B, which may be different values.

In FIG. 3H, the event in the store 100 has progressed. The man is stillstanding by the shelf and the woman has moved closer to the shelf.Camera clients 220A and 220B receive additional frames 320C and 320Dfrom cameras 305A and 305B. Camera client 220A again determines boundingareas 325C and 325D for the man and the woman, respectively, andcoordinates 322 for these bounding areas 325. Camera client 220Adetermines coordinates 322 (x₅, y₅) and (x₆, y₆) for bounding area 325Cand coordinates 322 (x₇, y₇) and (x₈, y₈) for bounding area 325D. Cameraclient 220A also generates a timestamp 324 that indicates that frame320C was received at time t₃. Camera client 220A generates frame data330C, indicating that frame 320C was generated by camera 305A andreceived by camera client 220A at t₃. Frame data 330C also indicatesthat the man corresponds to coordinates 322 (x₅, y₅) and (x₆, y₆) andhas a height at z₃ within frame 320C and that the woman corresponds tocoordinates 322 (x₇, y₇) and (x₈, y₈) and has a height at z₄ withinframe 320C.

Similarly, camera client 220B receives frame 320D from camera 305B.Camera client 220B determines bounding areas 325E and 325F for the manand the woman, respectively. Camera client 220B then determinescoordinates 322 (x₅, y₅) and (x₆, y₆) for bounding area 325E andcoordinates 322 (x₇, y₇) and (x₈, y₈) for bounding area 325F. Cameraclient 220B generates a timestamp 324 that indicates that frame 320D wasreceived at time t₄. Camera client 220B generates frame data 330D thatindicates frame 320D was generated by camera 305B and received by cameraclient 220B at t₄. Frame data 330D indicates that the man corresponds tocoordinates 322 (x₅, y₅) and (x₆, y₆) and has a height of z₃ in frame320D. Frame data 330D also indicates that the woman corresponds tocoordinates 322 (x₇, y₇) and (x₈, y₈) and has a height of z₄ withinframe 320D. Camera clients 220A and 220B communicate frame data 330C and330D to camera sever 225 when frame data 330C and 330D are ready.

In FIG. 3I, the event in the store 100 has further progressed and theman has removed an item 130 from the shelf. Camera client 220A receivesa frame 320E from camera 305A. Camera client 220A determines boundingareas 325G and 325H around the man and the woman, respectively. Cameraclient 220A determines coordinates 322 (x₉, y₉) and (x₁₀, y₁₀) forbounding area 325G and coordinates 322 (x₁₁, y₁₁) and (x₁₂, y₁₂) forbounding area 325H. Camera client 220A generates a timestamp 324indicating when frame 320E was received by camera client 220A (e.g., byusing internal clock 304). Camera client 220A generates frame data 330Ethat indicates that frame 320E was produced by camera 305A and receivedby camera client 220A at t₅. Frame data 330E indicates that the mancorresponds to coordinates 322 (x₉, y₉) and (x₁₀, y₁₀) and has a heightat z₅ within frame 320E. Frame data 330E also indicates that the womancorresponds to coordinates 322 (x₁₁, y₁₁) and (x₁₂, y₁₂) and has aheight at z₆ in frame 320E.

Camera client 220B receives frame 320F from camera 305B. Camera client220B determines bounding areas 325I and 325J around the man and thewoman, respectively. Camera client 220BA determines coordinates 322 (x₉,y₉) and (x₁₀, y₁₀) for bounding area 325I and coordinates 322 (x₁₁, y₁₁)and (x₁₂, y₁₂) for bounding area 325J. Camera client 220B generates atimestamp 324 indicating when frame 320F was received by camera client220B (e.g., by using internal clock 304). Camera client 220B thengenerates frame data 330F indicating that frame 320F was produced bycamera 305B and received by camera client 220B at t₆. Frame data 330Findicates that the man corresponds to coordinates 322 (x₉, y₉) and (x₁₀,y₁₀) and has a height at z₅ in frame 320F. Frame data 330F alsoindicates that the woman corresponds to coordinates 322 (x₁₁, y₁₁) and(x₁₂, y₁₂) and has a height at z₆ in frame 320F. Camera clients 220A and220B communicate frame data 330E and 330F to camera server 225 whenready.

4. Camera Server

FIGS. 3J-3P show the operation of camera server 225 in the camerasubsystem 202. Generally, camera server 225 receives frame data 330(e.g., 330A-330F) from the camera clients 220 in camera subsystem 202.Camera server 225 synchronizes and/or assigns the frame data 330 toparticular time windows 332 based on timestamps 324 in the frame data330. Camera server 225 then processes the information assigned toparticular time windows to determine the physical positions of peopleand/or objects within the space during those time windows 332.

In FIG. 3J, camera server 225 receives frame data 330 from the cameraclients 220 in camera subsystem 202. Camera server 225 assigns framedata 330 to time windows 332 depending on the timestamp 324 within framedata 330. Using the previous example, camera server 225 may determinethat timestamps 324 t₁, t₂, and t₃ fall within a first time window 322A(e.g., between times T0 and T1) and that timestamps 324 t₄, t₅, and t₆fall within a subsequent time window 332B (e.g., between times T1 andT2). As a result, camera server 225 assigns the frame data 330 forframes 320A, 320B, and 320C to time window 1 332A and the frame data 330for frames 320D, 320E, and 320F to time window 2 332B.

By assigning frame data 330 to time windows 332, camera server 225 mayaccount for desynchronization that occurs amongst the cameras 305,camera clients 220, and the camera server 225 in the camera subsystem202. The duration of the time windows 332 can be set to be larger thanthe desynchronization that is expected to occur to mitigate the effectsof desynchronization. For example, if the cameras 305 and camera clients220 are expected to desynchronize by a few milliseconds, then the timewindow 332 can be set to last 100 milliseconds to counteract thedesynchronization. In this manner, camera server 225 can mitigate theeffects of desynchronization as the camera subsystem 202 is scaled tohandle larger spaces by including more cameras 305 and camera clients220. In the example of FIG. 3J, camera server 225 sets the duration oftime window 1 332A to be between T0 and T1 and the duration of timewindow 2 332B to be between T1 and T2. Camera server 225 can set theduration of the time windows 332 to be any suitable amount to mitigatethe effects of desynchronization. In certain embodiments, T0 may be thetime when the cameras 305 in the camera subsystem 202 have finishedinitializing.

FIG. 3K shows an embodiment where camera server 225 uses cursors 335 toassign frame data 330 to time windows 332. Each cursor 335 maycorrespond to a particular camera client 220 in the camera subsystem202. In the example of FIG. 3K, cursor 335A corresponds to camera client1 220A, cursor 335B corresponds to camera client 3 220C, and cursor 335Ccorresponds to camera client 2 220B. Each cursor 335 points to aparticular time window 332. When frame data 330 is received from acamera client 220, that frame data 330 is generally assigned to the timewindow 332 to which the cursor 335 for that camera client 220 points.For example, if frame data 330 is received from camera client 1 220A,then that frame data 330 is generally assigned to time window 1 332A,because cursor 335A is pointing to time window 1 332A.

Camera server 225 may determine whether to advance cursor 335A whenframe data 330 is received from the camera client 220 corresponding tothat cursor 335. If that frame data 330 has a timestamp 324 that belongsin a subsequent time window 332, then camera server 225 may advance thecursor 335 to that time window 332, thereby indicating that cameraserver 225 is not expecting to receive any more frame data 330 from thatcamera client 220 that belongs in a prior time window 332. In thismanner, camera server 225 can quickly and efficiently assign frame data330 to time windows 332 without checking every time window 332 whenframe data 330 is received. For example, if camera client 2 220B isfaster at sending information than camera client 1 220A and cameraclient 3 220C, then cursor 335C may advance far ahead of cursors 335Aand 335B. When camera server 225 receives frame data 330 from cameraclient 2 220B, camera server 225 need not check every time window 332beginning from time window 1 332A to determine to which time window 332that frame data 330 should be assigned. Rather, camera server 225 canstart at the time window 332 to which cursor 335C points. In otherwords, camera server 225 need not first check whether a timestamp 324 inthe frame data 330 from camera client 2 220B indicates a time that fallswithin time window 1 332A and then whether that time falls within timewindow 2 332B. Instead, camera server 225 can first check whether thattime falls within time window 3 332C and ignore checking whether thattime falls within time window 1 332A and time window 2 332B. As aresult, the frame data 330 is quickly and efficiently assigned to thecorrect time window 332.

FIG. 3L illustrates camera server 225 moving out for processing framedata 330 that has been assigned to particular time windows 332.Generally, camera server 225 may determine that the frame data 330assigned to a particular time window 332 is ready for processing. Inresponse to that determination, camera server 225 may move the framedata 330 from a particular time window 332 to a task queue 336.Information in the task queue 336 is then processed to determine thephysical location of people or objects within a space during particulartime windows 332.

Camera server 225 determines that frame data 330 assigned to aparticular time window 332 is ready for processing in any suitablemanner. For example, camera server 225 may determine that a particulartime window 332 is ready for processing when that time window 332 hasframe data 330 for frames 320 from a sufficient number of cameras 305.Camera server 225 may use a threshold 338 to make this determination.When a particular time window 332 has been assigned frame data 330 forframes 320 from a number of cameras 305 that exceeds threshold 338,camera server 225 may determine that that time window 332 is ready forprocessing and move the information for that time window 332 to the taskqueue 336. For example, assume threshold 338 indicates that frame data330 for frames 320 from ten cameras 305 of an array 300 of twelvecameras 305 need to be received before a time window 332 is ready forprocessing. If time window 332 contains frame data 330 for frames 320from only eight cameras 305, then camera server 225 determines that timewindow 332 is not ready for processing, and as a result, time window 332waits to be assigned frame data 330 for frames 320 from additionalcameras 305. When time window 332 has received frame data 330 for frames320 from ten or more cameras 305, camera server 225 determines that timewindow 332 is ready for processing and moves frame data 330 in timewindow 332 to task queue 336.

Camera server 225 may also determine that a particular time window 332is ready for processing when a subsequent time window 332 has receivedframe data 330 for frames 320 from a number of cameras 305 exceedingthreshold 338. Using the previous example, even if time window 1 332Ahas been assigned frame data 330 for frames 320 from eight cameras,camera server 225 may nevertheless determine that time window 1 332A isready for processing when time window 2 332B has been assigned framedata 330 for frames 320 from ten or more cameras 305 (e.g., from everycamera 305 in camera array 300). In this scenario, camera server 225 mayassume that no additional frame data 330 will be assigned to time window1 332A because frame data 330 for frames 320 from a sufficient number ofcameras 305 has been assigned to a subsequent time window 2 332B. Inresponse, camera server 225 moves frame data 330 in time window 1 332Ato task queue 336.

Camera server 225 may also determine that a particular time window 332is ready for processing when that time window 332 has been awaitingprocessing for a certain period of time. For example, if an error or bugoccurs in the system and frames 320 from a number of cameras 305 are notsent or are lost, then a time window 332 may not receive frame data 330for frames 320 from enough cameras 305. As a result, processing for thattime window 332 may stall or be delayed. Camera server 225 may use atimeout or age-out beyond which a time window 332 does not wait forprocessing. Thus, when the time window 332 has not been processed for acertain period of time exceeding the timeout or the age-out, cameraserver 225 may nevertheless send the frame data 330 in that time window332 to the task queue 336. Using the previous example, assume thetimeout is 200 milliseconds. If time window 1 332A has been stuck withframe data 330 from frames 320 from eight cameras 305 for over 200milliseconds, camera server 225 may determine that time window 1 332Ahas waited long enough for additional frame data 330 and that timewindow 1 332A is ready for processing. In response, camera server 225moves frame data 330 in time window 1 332A to task queue 336.

In certain embodiments, when a time window 332 times out or ages out,camera server 225 may adjust threshold 338 so that future time windows332 are less likely to time out or age out. For example, camera server225 may lower threshold 338 when a time window 332 times out or agesout. Likewise, camera server 225 may increase threshold 338 when asubsequent time window 332 does not time out or age out. Camera server225 may adjust threshold 338 based on the number of cameras 305 thathave sent information for a particular time window 332. For example, ifa particular time window 332 times out or ages out when it has framedata 330 for frames 320 from eight cameras 305, and threshold 338 is tencameras 305, camera server 225 may reduce threshold 338 to a valuecloser to eight cameras. As a result, that time window 332 may then haveframe data 330 for frames 320 from a sufficient number of cameras 305and be moved to task queue 336. When a subsequent time window 332 doesnot time out because it has received frame data 330 for frames 320 fromnine cameras 305, camera server 225 may adjust threshold 338 towardsnine cameras 305. In this manner, camera server 225 may dynamicallyadjust the threshold 338 to prevent bugs, errors, and/or latency fromcausing delays in the camera subsystem 202.

In certain embodiments, camera server 225 processes time windows 332sequentially. In other words, camera server 225 does not process asubsequent time window 332 until a prior time window 332 is ready forprocessing. In the example of FIG. 3L, camera server 225 may not placetime window 2 332B into the task queue 336 until time window 1 332A hasbeen placed into the task queue 336. In this manner, the progression ofevents in a store 100 is evaluated sequentially (e.g., as the eventsunfold), which allows for proper tracking of the position of people inthe store 100. If time windows 332 were not evaluated sequentially, thenit may seem to the tracking system 132 that the event in the store 100progressed in a different and incorrect order.

FIG. 3M illustrates a task queue 336 of camera server 225. As shown inFIG. 3M, the task queue 336 includes frame data 330 from two timewindows 332. At the beginning of the task queue 336 is frame data 330for frames 320A, 320B, and 320C. Following in the task queue 336 isframe data 330 for frames 320D, 320E, and 320F. Camera server 225 mayprocess the entries in the task queue 336 in order. Thus, camera server225 may first process the first entry of the task queue 336 and processthe frame data 330 for frames 320A, 320B, and 320C. Camera server 225processes an entry of a task queue 336 and then moves that entry to aresult queue.

To process an entry of task queue 336, camera server 225 may combine orcluster the coordinates 322 of the same objects detected by the samecameras 320 to calculate combined coordinates 332 for that object. As aresult of this processing, each time window 332 should include only oneset of coordinates 322 per object per camera 305. After this processing,the combined coordinates 322 are placed into a result queue. FIG. 3Nillustrates a result queue 340 of camera server 225. As seen in FIG. 3N,result queue 340 includes the combined coordinates 332 for two timewindows 332.

As an example, camera server 225 first processes the first entry in thetask queue 336, which includes frame data 330 for frames 320A, 320B, and320C. Frames 320A and 320C are from the same camera 320A. As a result,camera server 225 may use the frame data 330A and 330C for frames 320Aand 320C to calculate a combined coordinate 322 for the people orobjects detected by camera 320A. As seen in FIG. 3N, camera server 225has determined combined coordinates 322 (x₁₃, y₁₃), and (x₁₄, y₁₄) and acombined height z₇ for object 1 detected by camera 1 305A and combinedcoordinates 322 (x₁₅, y₁₅) and (x₁₆, y₁₆) and a combined height z₈ forobject 2 detected by camera 1 305A. These combined coordinates 322 andcombined heights are the combined coordinates 322 and combined heightsfor the man and the woman in the video frames 302 received by camera305A during the first time window 332A. Likewise, camera server 225 maydetermine combined coordinates 322 and combined heights for the objectsdetected by camera 2 305B during the first time window 332A. Forexample, camera server 225 may use frame data 330B for frame 320B (andframe data 330 for any other frames 320 received by camera 2 305B duringthe first time window 332A) to determine combined coordinates 322 (x₁₃,y₁₃), and (x₁₄, y₁₄) and a combined height z₇ for object 1 detected bycamera 2 305B and combined coordinates 322 (x₁₅, y₁₅) and (x₁₆, y₁₆) anda combined height z₈ for object 2 detected by camera 2 305B. Cameraserver 225 may determine combined coordinates 322 for each objectdetected by cameras 305 in the first time window 332A in this manner.

Camera server 225 then determines combined coordinates 322 for objectsdetected by the cameras 305 during the second time window 332B in asimilar fashion. For example, camera server 225 may use frame data 330Efor frame 320E (and frame data 330 for any other frames 320 received bycamera 1 305A during the second time window 332B) to determine combinedcoordinates 322 (x₁₇, y₁₇), and (x₁₈, y₁₈) and a combined height z₉ forobject 1 detected by camera 1 305A and combined coordinates 322 (x₁₉,y₁₉) and (x₂₀, y₂₀) and a combined height z₁₀ for object 2 detected bycamera 1 305A. Camera server 225 may also use frame data 330D and 330Ffor frames 320D and 320F to determine combined coordinates 322 (x₁₇,y₁₇), and (x₁₈, y₁₈) and a combined height z₉ for object 1 detected bycamera 2 305B and combined coordinates 322 (x₁₉, y₁₉) and (x₂₀, y₂₀) anda combined height z₁₀ for object 2 detected by camera 2 305B.

Camera server 225 calculates combined coordinates 322 and combinedheights in any suitable manner. For example, camera server 225 maycalculate combined coordinates 322 and combined heights by taking theaverage of the coordinates 322 and the heights of particular objectsdetected by the same camera 305 in a particular time window 332. Usingthe example in FIG. 3M, camera server 225 may calculate combinedcoordinates 322 (x₁₃, y₁₃) for camera 1 305A by taking the average ofcoordinates 322 (x₁, y₁) and (x₅, y₅) from frame data 330A and 330C.Similarly, camera server 225 may determine the combined coordinate 322(x₁₄, y₁₄) for camera 1 305A by taking the average of coordinates 322(x₂, y₂) and (x₆, y₆) from frame data 330A and 330C. Camera server 225may determine combined height z₇ for camera 1 305A by taking the averageof heights z₁ and z₃ from frame data 330A and 330C. Similarly, cameraserver 225 may determine combined coordinates 322 (x₁₇, y₁₇) for camera2 305B by taking the average of coordinates 322 (x₅, y₅) and (x₉, y₉)from frame data 330D and 330F. Likewise, camera server 225 may determinecombined coordinates 322 (x₁₈, y₁₈) for camera 2 305B by taking theaverage of coordinates 322 (x₆, y₆) and (x₁₀, y₁₀) from frame data 330Dand 330F. Camera server 225 may determine combined height z₉ for camera2 305B by taking the averages of heights z₃ and z₅ from frame data 330Dand 330F. Camera server 225 takes these averages because these are thecoordinates 322 and heights for the same object determined by the samecamera 305 during the same time window 332.

Camera server 225 may follow a similar process to determine or tocalculate the combined coordinates for object 2 detected by cameras 1305A and 2 305B.

Camera server 225 may calculate combined coordinates 322 (x₁₅, y₁₅) forcamera 1 305A by taking the average of coordinates 322 (x₃, y₃) and (x₇,y₇) from frame data 330A and 330C. Similarly, camera server 225 maydetermine the combined coordinate 322 (x₁₆, y₁₆) for camera 1 305A bytaking the average of coordinates 322 (x₄, y₄) and (x₈, y₈) from framedata 330A and 330C. Camera server 225 may determine combined height z₈for camera 1 305A by taking the average of heights z₂ and z₄ from framedata 330A and 330C. Similarly, camera server 225 may determine combinedcoordinates 322 (x₁₉, y₁₉) for camera 2 305B by taking the average ofcoordinates 322 (x₇, y₇) and (x₁₁, y₁₁) from frame data 330D and 330F.Likewise, camera server 225 may determine combined coordinates 322 (x₂₀,y₂₀) for camera 2 305B by taking the average of coordinates 322 (x₈, y₈)and (x₁₂, y₁₂) from frame data 330D and 330F. Camera server 225 maydetermine combined height z₁₀ for camera 2 305B by taking the averagesof heights z₄ and z₆ from frame data 330D and 330F.

Camera server 225 uses any other suitable calculation to calculatecombined coordinates and combined heights. For example, camera server225 may take a median of coordinates 322 and heights for objectsdetected by the same camera 305 during a time window 332. Camera server225 may also use clustering processes to calculate the combinedcoordinates 322 and combined heights. For example, camera server 225 mayuse K-means clustering, Density-based spatial clustering of applicationswith noise (DBSCAN), k-medoids, gaussian mixture models, andhierarchical clustering to calculate combined coordinates 322 andcombined heights.

After camera server 225 has calculated the combined coordinates 322 andcombined heights, camera server 225 has determined the coordinates 322for each object detected by each camera 305 during a time window 332.However, camera server 225 may perform additional processing todetermine whether the object detected by different cameras 305 are thesame object. Camera server 225 may use linking and homography todetermine which objects detected by which cameras 305 are actually thesame person or object in a space. Camera server 225 may then take thecombined coordinates 322 for those objects from the different cameras305 and employ homography to determine a physical location for thatperson or object in the physical space during a time window 332.Embodiments of this process are described in U.S. Patent Application No.______ entitled, “Topview Object Tracking Using a Sensor Array”(attorney docket no. 090278.0180), the contents of which areincorporated by reference herein in its entirety. In this manner, cameraserver 225 determines the physical locations of people and/or objectswithin the space during particular time windows 332.

In particular embodiments, camera clients 220 may also use the same timewindows 332 as camera server 225 to communicate frame data 330 inbatches to camera server 225. As seen in FIG. 30, camera client 220assigns frame date 330 to time windows 332 based on the timestamps 324within that frame data 330. Camera client 220 may determine that aparticular time window 332 is ready to be communicated to camera server225 in a similar way as camera server 225 determines a time window 332is ready for processing. When camera client 220 determines that aparticular time window 332 is ready (e.g., when each camera 305communicatively coupled to camera client 220 has communicated a frame inthat time window 332), camera client 220 communicates the frame data 330assigned to that time window 332 as a batch to the camera server 225. Inthis manner, camera server 225 may assign frame data 330 to time windows332 even more quickly and more efficiently because camera server 225receives the frame data 330 for a time window 332 as a batch from cameraclient 220.

In certain embodiments, even if camera server 225 and camera clients 220are not synchronized, camera server 225 can account fordesynchronization that occurs (e.g., by desynchronized internal clocks302, by latency differences between camera clients 220 to camera server225, by processing speed differences between camera clients 220, etc.)by adjusting the timestamps 324 in frame data 330. FIG. 3P shows cameraserver 225 adjusting timestamps 324. As discussed previously, frame data330 includes a timestamp 324 generated by camera client 220 thatindicates when camera client 220 received a frame 320. In the example ofFIG. 3P, frame data 330 indicates that camera client 220 received frame320 at time t₁. If the camera clients 220 and camera server 225 are notsynchronized, then the timestamp 324 t₁ is relatively meaningless tocamera server 225 because the camera server 225 cannot be assured thattimestamps 324 from different camera clients 220 are accurate relativeto each other. Thus, it is difficult, if not impossible, to preciselyanalyze frame data 330 from different and/or multiple camera clients220.

Camera server 225 can adjust timestamps 324 for particular cameras 305to account for desynchronization. Generally, camera server 225determines a delay for each camera 305 by tracking the delay for priorframes 320 from that camera 305. Camera server 225 then adjuststimestamps 324 for frame data 330 for frames 320 from that camera 305 bythe determined delay. In the example of FIG. 3P, camera server 225determines a delay for camera 1 305A by determining, for each frame 320(x) from camera 1, the difference in time (labeled Δ_(x)) between thetimestamp 324 indicated in frame data 330 for that frame 320 (labeledt_(x)) and the time camera server 225 received the frame data 330(labeled T_(x)). Camera server 225 calculates an average delay (labeledΔ) by averaging the differences in time (Δ_(x)) for a prior number offrames 320. In the example of FIG. 3P, camera server 225 averages thedifferences in time for the previous thirty frames 320 to determine theaverage delay. Camera server then adds the average delay (Δ) to thetimestamp 324 for the frame data 330 to adjust the timestamp 324 toaccount for desynchronization. In this manner, camera server 225 andtracking system 132 can function properly even if camera clients 220 andcamera server 225 are not synchronized (e.g., according to a clocksynchronization protocol).

5. Example Method

FIGS. 3Q and 3R are flowcharts illustrating an example method 342 ofoperating the camera subsystem 202. In particular embodiments, variouscomponents of the camera subsystem 202 perform the steps of method 342.Generally, by performing method 342, the camera subsystem 202 determinesthe physical position of people or objects within a space.

As seen in FIG. 3Q, method 342A begins with cameras 305A and 305Bgenerating and communicating frames 320A and 320D to camera clients 220Aand 220B, respectively. Camera clients 220A and 220B then determinecoordinates 322 for two people detected in frames 320A and 320B. Thesecoordinates may define bounding areas 325 around these people.

Camera 305A then generates frame 320B and communicates frame 320B tocamera client 220A. Camera client 220A generates coordinates 322 for twopeople shown in frame 320B. During that process, camera 305B generatesframe 320E and communicates frame 320E to camera client 220B. Cameraclient 220B then determines coordinates 322 for two people detected inframe 320E. Camera 305A then generates frame 320C and communicates frame320C to camera client 220A. Camera client 220A determines coordinates322 for two people detected in frame 320C. Importantly, FIG. 3Q showsthat frames from cameras 305A and 305B may not be generated andcommunicated simultaneously or synchronously. Additionally, coordinatesfor people detected in frames 320 may not be generated simultaneously orsynchronously in camera clients 220A and 220B.

FIG. 3R shows method 342B which continues from method 342A of FIG. 3Q.As seen in FIG. 3R, camera client 220A generates frame data 330 from thecoordinates 322 for the two people detected in frame 320A. Likewise,camera client 220B generates frame data 330 using the coordinates 322for the two people detected in frame 320D. Camera clients 220A and 220Bcommunicate the frame data 330 to camera server 225. Camera client 220Agenerates additional frame data 330 using the coordinates 322 for thetwo people detected in frame 320B. Camera client 220A then communicatesthat frame data 330 to camera server 225. Camera server 225 may assignthe frame data 330 to a time window 332. Camera server 225 may determinethat that time window 332 is ready for processing in step 344 and, inresponse, place the frame data 330 in that time window 332 into a taskqueue 336 in step 346. Camera server 225 may then combine or cluster thecoordinates 322 in that time window 322 to determine combinedcoordinates 322 in step 348. For example, camera server 225 may averagethe coordinates 322 in that time window to determine combinedcoordinates 322 for the people detected by the different cameras 305during that time window 332. Camera server 225 may then map the peopledetected by the different cameras 305 to people in the space in step350. Camera server 225 may then determine the positions of the peopleduring that time window 332 in step 352. Camera server 225 communicatesthese determined positions to central server 240.

Modifications, additions, or omissions may be made to method 342depicted in FIGS. 3Q and 3R. Method 342 may include more, fewer, orother steps. For example, steps may be performed in parallel or in anysuitable order. While discussed as particular components of camerasubsystem 202 performing the steps, any suitable component of camerasubsystem 202 may perform one or more steps of the method.

6. Other Features

In particular embodiments, the camera subsystem 202 may include a secondcamera array that operates in tandem with the first camera array 300 ofthe camera subsystem 202. FIG. 3S shows an embodiment that includes twocamera arrays 300 and 354. Camera array 300 includes cameras 305M.Camera array 354 includes cameras 305N. Cameras 305N operate in the sameway as cameras 305M and can be used to determine positions of objectsand/or people in a space using the same techniques described using FIGS.3A-3R.

Each camera 305N is positioned slightly offset from a camera 305M ofcamera array 300. In this manner, cameras 305M capture video that issimilar to the video captured by cameras 305N. In certain embodiments,cameras 305M may use different versions of software or differentversions of software may be used to process video from cameras 305Mrelative to cameras 305N. In this manner, newer software can be run forcameras 305N to test the effectiveness of that software. The testing ofthat software does not interrupt the operation of the camera subsystem202 because cameras 305M may still be using the previous software, whichalso acts as a baseline for comparing against the operation of the newsoftware running on cameras 305N. For example, the accuracy of theposition tracking provided by the new software can be determined andcompared against the accuracy provided by the old software. If the newsoftware is less accurate than the old software, then the old softwareshould continue to be used.

In certain embodiments, camera server 225 can retrieve video footagefrom camera clients 220 or a shared memory if the camera server 225 isunable to determine the positions of people based on the frame data 330from the camera clients 220. FIG. 3T shows a camera server 225retrieving videos 302 from camera clients 220 and/or shared memory 356.Generally, camera clients 220 store video received from cameras locallyor in a shared memory 356. That video 302 is then made available tocamera server 225 if camera server 225 cannot determine the positions ofpeople based on frame data 330. Camera server 225 may analyze video 302to determine the positions of people in the space. Camera server 225 mayperform better and more accurate analysis of the raw video footage thancamera clients 220, and thus, camera server 225 may generate moreaccurate frame data 330 than camera clients 220. In some embodiments,camera server 225 may have frame data 330 from one camera client 220that conflicts or does not align with frame data 330 from another cameraclient 220. Camera server 225 can retrieve the raw video footage todetermine which frame data 330 should be accepted and used.

In the example of FIG. 3T, camera client 220A stores video 302A locallyor in shared memory 356. Camera client 220B stores video 302B locally orin shared memory 356. When camera server 225 is unable to determine thepositions of people based on frame data 330, camera server 225 sends arequest 358 to camera client 220A and/or shared memory 356. In response,camera client 220A and/or shared memory 356 send video 302A to cameraserver 225. Camera server 225 may then analyze the video 302A todetermine the positions of people in the space.

III. Light Detection and Ranging (LiDAR) Subsystem

Certain embodiments of tracking system 132 include a LiDAR subsystem204. FIGS. 4A-4D show the LiDAR subsystem 204 and its operation withintracking system 132. Generally, LiDAR subsystem 204 uses LiDAR sensorsand a LiDAR server to track the positions of people and/or objectswithin a physical space. LiDAR subsystem 204 may be used on its own orin conjunction with other subsystems (e.g., camera subsystem 202) totrack the positions of people and/or objects in the space.

FIG. 4A shows an example LiDAR subsystem 204. As seen in FIG. 4A, LiDARsubsystem 204 includes a LiDAR array 400 and a LiDAR server 230.Generally, LiDAR sensors 405 in LiDAR array 400 detect the presence ofpeople and/or objects within a space and determine coordinates for thesepeople and/or objects. LiDAR server 230 processes these coordinates todetermine the physical positions of the people and/or objects in thespace.

LiDAR array 400 is an array of LiDAR sensors 405. LiDAR array 400 may bepositioned above a physical space to detect the presence and positionsof people and/or objects within the space. In the example of FIG. 4A,LiDAR array 400 is a 3×4 array of LiDAR sensors 405. LiDAR array 400includes any suitable number of LiDAR sensors 405 arranged in an arrayof any suitable dimensions.

Each LiDAR sensor 405 detects the presence of people and/or objectswithin a portion of the physical space. Generally, LiDAR sensors 405emit light pulses into the space. These light pulses are reflected backtowards the LiDAR sensors 405 when the light pulses contact peopleand/or objects in the space. The LiDAR sensor 405 tracks characteristicsof the reflected light pulses, such as the return times of the lightpulses and the wavelength of the return light pulses, to detect thepresence of people and/or objects within the physical space. LiDARsensors 405 may also determine coordinates for the detected peopleand/or objects. LiDAR sensors 405 communicate the coordinates for thedetected people and/or objects to LiDAR server 230.

LiDAR sensors 405 may be communicatively coupled to LiDAR server 230 inany suitable manner. For example, LiDAR sensors 405 may be hardwired toLiDAR server 230. As another example, LiDAR sensors 405 may wirelesslycouple to LiDAR server 230 using any suitable wireless standard (e.g.,WiFi). LiDAR sensors 405 communicate coordinates for detected peopleand/or objects through the communication medium to LiDAR server 230.

FIG. 4B shows a LiDAR sensor 405 communicating coordinates 410 to LiDARserver 230. Generally, LiDAR sensor 405 analyzes characteristics ofreflected light pulses to determine the coordinates 410 of people and/orobjects within the space. LiDAR sensor 405 communicates thesecoordinates 410 to LiDAR server 230 for further processing. In theexample of FIG. 4B, LiDAR sensor 405 detects coordinates 410 for atleast two people and/or objects in the space. The coordinates 410 forthese people and/or objects are (x₁, y₁) and (x₂, y₂). LiDAR sensor 405communicates these coordinates 410 to LiDAR server 230 for furtherprocessing.

FIG. 4C illustrates the general operation of LiDAR server 230. As seenin FIG. 4C, LiDAR server 230 processes coordinates 410 received from theLiDAR sensors 405. LiDAR server 230 assigns coordinates 410 to timewindows 332 in a similar manner as camera server 225 assigns frame data330 to time windows 332. For example, LiDAR server 230 may assigncoordinates 410 to particular time windows 332 based on the time thatLiDAR server 230 received the coordinates 410 from LiDAR sensor 405.

LiDAR server 230 may process the coordinates 410 assigned to a timewindow 332 to determine the physical position of people and/or objectswithin the space. In the example of FIG. 4C, LiDAR server 230 receivescoordinates 410 for two people from two different LiDAR sensors 405. OneLiDAR sensor 405 provides coordinates 410 (x₁, y₁) and (x₂, y₂) for thetwo people, respectively. Another LiDAR sensor 405 provides coordinates410 (x₁, y₁) and (x₂, y₂) for the same two people, respectively. As withcamera client 220 and camera server 225, the subscripts on thesecoordinates 410 are not meant to indicate that these coordinates 410have the same value, but, rather, that these are the first and secondcoordinates 410 provided by a particular LiDAR sensor 405.

LiDAR server 230 uses these coordinates 410 to determine the physicalposition of people within the space. As with the camera server 225,LiDAR server 230 may determine that the coordinates 410 provided by twodifferent LiDAR sensors 405 correspond to the same person within thephysical space. In response, LiDAR server 230 may take these coordinates410 and use homography to determine a position of the person within thephysical space in a particular time window 332. In the example of FIG.4C, LiDAR server 230 uses coordinates 410 to determine the position of afirst person during the time window 332 to be (x₃, y₃). LiDAR server 230also uses coordinates 410 to determine the physical position of a secondperson during the time window 332 to be (x₄, y₄). LiDAR server 230communicates these physical positions to central server 240 for furtherprocessing.

FIG. 4D shows a method 415 for the operation of the LiDAR subsystem 204in the tracking system 132. Generally, LiDAR subsystem 204 performsmethod 415 to determine the positions of people and/or objects within aphysical space.

LiDAR sensor 405 determines coordinates 410 of detected people andcommunicates these coordinates 410 to LiDAR server 230. LiDAR sensor 405may determine these coordinates 410 by emitting a light pulse andanalyzing characteristics of the light pulse when that light pulse isreflected back to LiDAR sensor 405. For example, LiDAR sensor 405 mayanalyze the return time of the reflected light pulse and/or thewavelength of the reflected light pulse to determine whether a person ispresent in the physical space and the coordinates 410 of that person.

LiDAR server 230 analyzes the coordinates 410 from LiDAR sensor 405 todetermine the positions of people within the physical space during afirst time window 332 in step 416. LiDAR server 230 then communicatesthese positions to central server 240. LiDAR sensor 405 may subsequentlydetermine the coordinates 410 of detected people and communicate thesecoordinates 410 to LiDAR server 230. LiDAR server 230 may againdetermine the positions of these people in a subsequent time window 332and communicate these positions to central server 240 in step 418.

As with the camera subsystems 202, central server 240 may use thesepositions to determine which person removed an item 130 from the spaceduring the particular time window 332. The operation of central server240 will be described in more detail using FIG. 6A through FIG. 6C.

Modifications, additions, or omissions may be made to method 415depicted in FIG. 4D. Method 415 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as components of LiDAR subsystem 204 performingthe steps, any suitable component of tracking system 132, such ascentral server 240 for example, may perform one or more steps of themethod.

IV. Weight Subsystem

Tracking system 132 includes a weight subsystem 206 that includes weightsensors 215 and weight server 235. Generally, weight sensors 215 detectthe weights of items positioned above or near the weight sensors 215.The weight sensors 215 may be positioned on an unconventional rack 115that holds items. Weight server 235 tracks the weights detected byweight sensors 215 to determine if and when items 130 are removed fromthe rack 115. The weight sensors 215, rack 115, and weight server 235will be described in more detail using FIGS. 5A-5J.

FIG. 5A illustrates an example weight sensor 500 of weight subsystem206. As seen in FIG. 5A, weight sensor 500 includes plates 510A and510B, load cells 505A, 505B, 505C, and 505D, and wires 515A, 515B, 515C,515D, and 520. Generally the components of weight sensor 500 areassembled so that weight sensor 500 can detect a weight of items 130positioned above or near weight sensor 500.

Plates 510 form surfaces that distribute the weight of items 130 acrossthe surfaces. Plates 510 may be made of any suitable material, such as,for example, metal and/or plastic. Items 130 may be positioned above ornear plates 510 and the weight of these items 130 may be distributedacross plates 510.

Load cells 505 are positioned between plates 510A and 510B. Load cells505 produce electrical signals based on the weight experienced by theload cells 505. For example, load cells 505 may be transducers thatconverts an input mechanical force (e.g., weight, tension, compression,pressure, or torque) into an output electrical signal (e.g., current orvoltage). As the input force increases, the output electrical signal mayincrease proportionally. Load cells 505 may be any suitable type of loadcell (e.g., hydraulic, pneumatic, and strain gauge). Although load cells1310 are illustrated as being cylindrical in shape, they may be anysuitable size and shape that is appropriate for the particularimplementation contemplated.

The signals from load cells 505 may be analyzed to determine an overallweight of items 130 positioned above or near weight sensor 500. Loadcells 505 may be positioned such that the weight of items 130 positionedabove or near weight sensor 500 is evenly distributed to each load cell505. In the example of FIG. 5A, load cells 505 are positionedsubstantially equidistant from corners of plates 510A and 510B. Forexample, load cell 505A is positioned a distance d1 from a corner ofplates 510A and 510B. Load cell 505B is positioned a distance d2 from acorner of plates 510A and 510B. Load cell 505C is positioned a distanced3 from a corner of plates 510A and 510B. Load cell 505D is positioned adistance d4 from a corner of plates 510A and 510B. Distances d1, d2, d3and d4 may be substantially equal to each other. This disclosurecontemplates distances differing by 5 to 10 millimeters and still beingconsidered substantially equal to each other. By positioning load cells505 substantially equal distances from corners of plates 510A and 510B,the weight of items positioned above or near weight sensor 500 is evenlydistributed across the load cells 505. As a result, the total weight ofitems positioned above or near weight sensor 500 can be determined bysumming the weights experienced by the individual load cells 505.

Load cells 505 communicate electric signals that indicate a weightexperienced by the load cells 505. For example, the load cells 505 mayproduce an electric current that varies depending on the weight or forceexperienced by the load cells 505. Each load cell 505 is coupled to awire 515 that carries the electric signal. In the example of FIG. 5A,load cell 505A is coupled to wire 515A; load cell 505B is coupled towire 515B; load cell 505C is coupled to wire 515C; and load cell 505D iscoupled to wire 515D. Wires 515 are grouped together to form wire 520that extends away from weight sensor 500. Wire 520 carries the electricsignals produced by load cells 505 to a circuit board that communicatesthe signals to weight server 235.

Weight sensor 500 may be disposed in an unconventional rack 115 designedto hold items. FIG. 5B shows an example rack 525. As seen in FIG. 5B,rack 525 includes a base 530, one or more panels 535, and one or moreshelves 540. Generally, base 530 is at the bottom of rack 525 and formsa foundation for the other components of rack 525. Panels 535 extendvertically upwards from base 530. Shelves 540 couples to panels 535and/or base 530. For example, two shelves 540 may couple to a panel 535and extend away from panel 535. Generally, panels 535 and base 530 allowshelves 540 to hold the weight of items positioned on shelves 540.Weight sensors 500 may be disposed within shelves 540 to detect theweight of items positioned on shelf 540.

FIG. 5C shows an exploded view of rack 525. As seen in FIG. 5C, base 530is formed using several surfaces 532. Surface 532A forms a bottomsurface of base 530. Surfaces 532B and 532D form the sides of base 530.Surface 532C forms a back surface of base 530. Surface 532E forms a topsurface of base 530. This disclosure contemplates base 530 being formedusing any suitable materials such as, for example, wood, metal, glass,and/or plastic. Surface 532A may be coupled to surfaces 532B, 532C, and532D. Surface 532B may be coupled to surfaces 532A, 532E, and 532C.Surface 532C may be coupled to surfaces 532A, 532B, 532D, and 532E.Surface 532D may be coupled to surfaces 532A, 532C, and 532E. Surface532E may be coupled to surfaces 532B, 532C, and 532D. Surfaces 532B,532C, and 532D extend upwards from surface 532A. Generally, surfaces532A, 532B, 532C, 532D, and 532E form a box structure around a space542. Base 530 includes a drawer 545 that can open to allow access intothat space 542. Drawer 545 is positioned within the space 542. Whendrawer 545 is closed, base 530 may form an enclosure around the space542. When drawer 545 is open, access to the space 542 may be providedthrough the open drawer 545. In certain embodiments, a door may be usedto provide access to space 542 rather than drawer 545.

Surface 532E defines a cavity 534 that also allows access into the space542. Generally, cavity 534 allows wires 520 from weight sensors 500 toextend into the space 542.

Panels 535 extend upwards from base 530. Panels 535 may be formed usingany suitable materials, such as for example, wood, metal, and/orplastic. As seen in FIG. 5C, panels 535 define one or more cavities 550that extend along the width of panels 535. Cavities 550 allow wires 520from weight sensors 500 to extend into a space 552 defined by panels535. Generally, space 552 is a hollow interior of panel 535. Wires 520extend through cavity 550 and down space 552 towards cavity 534. In thismanner, wires 520 may be run from weight sensors 500 down to space 542in base 530. Each cavity 550 may correspond to a shelf 540 that couplesto panel 535.

Each shelf 540 couples to panel 535 and/or base 530. Weight sensors 500are disposed in the shelf 540. A shelf 540 may couple to panel 535 suchthat the wires 520 of the weight sensors 500 disposed in the shelf 540can run from the weight sensors 500 through a cavity 550 into space 552.These wires 520 then run down space 552 and through cavity 534 intospace 542.

FIGS. 5D and 5E illustrate an example shelf 540. FIG. 5D shows a frontview of shelf 540. As seen in FIG. 5D, shelf 540 includes a bottomsurface 560A, a front surface 560B, and a back surface 560C. Bottomsurface 560A is coupled to front surface 560B and back surface 560C.Front surface 560B and back surface 560C extend upwards from bottomsurface 560A. Multiple weight sensors 500 are positioned on bottomsurface 560A between front surface 560B and back surface 560C. Eachweight sensor 500 is positioned to detect a weight of items 130positioned within certain regions 555 of shelf 540. Each region 555 maybe designated using dividers 558. Items placed within a particularregion 555 will be detected and weighed by the weight sensor 500 forthat region 555. This disclosure contemplates shelf 540 being made usingany suitable material such as, for example, wood, metal, glass, and/orplastic. Wires 515 and 520 have not been illustrated in FIG. 5D so thatthe structure of shelf 540 can be shown clearly, but their omission fromFIG. 5D should not be interpreted as their removal. This disclosurecontemplates that wires 515 and 520 are present and connected to weightsensors 500 in the example of FIG. 5D.

FIG. 5E shows a back view of shelf 540. As seen in FIG. 5E, back surface560C defines a cavity 562. Wires 520 of weight sensors 500 extend fromthe weight sensors 500 through cavity 562. Generally, back surface 560Cof shelf 540 is coupled to panel 535 such that cavity 562 is at leastpartially aligned with cavity 550 in the panel 535. In this manner,wires 520 can run from weight sensors 500 through cavity 562 and throughcavity 550.

In certain embodiments, weight sensor 500 is positioned in shelf 540such that weight sensor 500 detects the weight of items positionedwithin a particular region 555 of shelf 540. As seen in the examples ofFIGS. 5D and 5E, shelf 540 includes four regions 555 that are positionedabove four weight sensors 500. Each weight sensor 500 detects the weightof items positioned within their corresponding regions 555. Due to thepositioning of weight sensors 500, a weight sensor 500 may not beaffected by the weight of items 130 positioned in regions 555 that donot correspond to that weight sensor 500.

FIG. 5F shows an example base 530. As seen in FIG. 5F, base 530 may alsoaccommodate weight sensors 500. For example, weight sensors 500 may bepositioned on a top surface 532E of base 530. Wires 520 for these weightsensors 500 may run from the weight sensors 500 through cavity 534 intospace 542. As a result, items may be positioned on base 530 and theirweights may be detected by weight sensors 500.

A circuit board 565 is positioned in space 542. Circuit board 565includes ports to which wires 520 from the weight sensors 500 of rack525 connect. In other words, circuit board 565 connects to wires 520from weight sensors 500 positioned on base 530 and on shelves 540. Thesewires 520 enter space 542 through cavity 534 and connect to circuitboard 565. Circuit board 565 receives the electric signals produced bythe load cells 505 of the weight sensors 500. Circuit board 565 thencommunicates signals to weight server 235 indicating the weightsdetected by the weight sensors 500. Drawer 545 may open to allow accessto space 542 and to circuit board 565. For example, drawer 545 may beopened so that circuit board 565 may be serviced and/or repaired.

FIG. 5G shows an example circuit board 565. As seen in FIG. 5G, circuitboard 565 includes a processor 566 and multiple ports 568. Generally,ports 568 couple to wires 520 from weight sensors 500. This disclosurecontemplates circuit board 565 including any suitable number of ports568 to connect to the wires 520 from the weight sensors 500 of rack 525.Processor 566 receives and processes the signals from ports 568.

Circuit board 565 may communicate signals to weight server 235 throughany suitable medium. For example, circuit board 565 may communicatesignals to weight server 230 through an ethernet connection, a wirelessconnection (e.g., WiFi), a universal serial bus connection, and/or aBluetooth connection. Circuit board 565 can automatically select aconnection through which to communicate signals to weight server 235.Circuit board 565 may choose the connection based on priority. Forexample, if the ethernet connection is active, circuit board 565 mayselect the ethernet connection for communicating with weight server 235.If the ethernet connection is down and the wireless connection isactive, circuit board 565 may choose the wireless connection tocommunicate with weight server 235. If the ethernet connection and thewireless connection are down and the universal serial bus connection isactive, circuit board 565 may select the universal serial bus connectionto communicate with weight server 235. If the ethernet connection, thewireless connection, and the universal serial bus connection are downand the Bluetooth connection is active, circuit board 565 may select theBluetooth connection to communicate with weight server 235. In thismanner, circuit board 565 has improved resiliency because circuit board565 may continue to communicate with weight server 235 even if certaincommunication connections go down.

Circuit board 565 may receive electrical power through variousconnections. For example, circuit board 565 may include a power port 570that supplies electrical power to circuit board 565. An electrical cablethat plugs into an electrical outlet may couple to power port 570 tosupply electrical power to circuit board 565. Circuit board 565 may alsoreceive electrical power through the ethernet connection and/or theuniversal serial bus connection.

FIG. 5H shows a signal 572 produced by the weight sensor 500. As seen inFIG. 5H the signal 572 begins by indicating a certain weight detected bythe weight sensors 500. Around time t₁ an item positioned above theweight sensor 500 is taken. As a result, the weight sensor 500 detects adrop in the weight and the signal 572 experiences a corresponding drop.Beyond time t₁, the signal 572 continues to hover around the lowerweight because the item 130 was removed. This disclosure contemplatesthat the signal 572 may include noise introduced by the environment suchthat the signal 572 is not a perfectly straight or smooth signal.

FIG. 5I shows an example operation of weight server 235. As seen in FIG.5I, weight server 235 receives a signal 572 from a weight sensor 500 attime to indicating a weight w₀. Similar to camera server 225, weightserver 235 may assign this information to a particular time window 332Abased on the indicated time of to. Later, weight server 235 may receivea signal 572 from the weight sensor 500 indicating that at time t₁, anew weight w₁ is detected. Weight w₁ may be less than weight w₀, therebyindicating that an item 130 may have been removed. Weight server 235assigns the information to a subsequent time window 332C based on thetime indicated at t₁.

Weight server 235 may implement an internal clock 304E that issynchronized with the internal clocks 304 of other components oftracking system 132 (e.g., camera clients 220, camera server 225, andcentral server 240). Weight server 235 may synchronize the internalclock 304E using a clock synchronization protocol (e.g., Network TimeProtocol and/or Precision Time Protocol). Weight server 235 may useclock 304E to determine the times at which signals 572 from weightsensors 500 were received and assign these signals 572 to theirappropriate time windows 332.

In certain embodiments, time windows 332 in weight server 235 arealigned with time windows 332 in camera clients 220, camera server 225,and/or central server 240. For example, time window 332A in weightserver 235 may have the same start time (T0) and end time (T1) as timewindow 332A in camera server 225 in the example of FIG. 3J. In thismanner, information from different subsystems of tracking system 132 maybe grouped according to the same time windows 332, which allows thisinformation to be correlated to each other in time.

Similar to camera server 225, weight server 235 may process theinformation in the time windows 332 sequentially when the time windows332 are ready for processing. Weight server 235 may process theinformation in each time window 332 to determine whether an item 130 wasremoved during that particular time window 332. In the example of FIG.51, when weight server 235 processes the third time window 332C, weightserver 235 may determine that sensor 1 500 detected that two items weretaken during time window 3 332C; thereby, resulting in the weight dropfrom w₀ to w₁. Weight server 235 may make this determination bydetermining a difference between w₀ and w₁. Weight server 235 may alsoknow (e.g., through a lookup table) the weight of an item 130 positionedabove or near weight sensor 500. Weight server 235 may divide thedifference between w₀ and w₁ to determine the number of items 130removed. Weight server 235 may communicate this information to centralserver 240 for further processing. Central server 240 may use thisinformation along with the tracked positions of people within the spaceto determine which person in the space removed the items 130.

FIG. 5J shows an example method 580 for operating weight subsystem 206.Generally, various components of weight subsystem 206 perform method 580to determine when certain items 130 were taken.

Weight sensor 215 detects the weight experienced 582 above or aroundweight sensor 215 and communicates the detected weight 582 through anelectric signal 572 to weight server 235. Weight server 235 may analyzethe signals 572 from weight sensor 215 to determine a number 584 ofitems 130 that were taken during a first time window 332. Weight server235 may communicate the determination to central server 240. Weightsensor 215 may subsequently detect a weight 586 experienced by weightsensor 215 and communicate that weight 586 to weight server 235. Weightserver 235 may analyze that weight 586 to determine a number 588 ofitems 130 that were taken during a second time window 332. Weight server235 may communicate that determination to central server 240. Centralserver 240 may track whether items 130 were taken during particular timewindows 332. And if so, central server 240 may determine which person inthe space took those items 130.

Modifications, additions, or omissions may be made to method 580depicted in FIG. 5J. Method 580 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as various components of weight subsystem 206performing the steps, any suitable component of tracking system 132,such as central server 240 for example, may perform one or more steps ofthe method.

V. Central Server

FIGS. 6A-6C show the operation of central server 240. Generally, centralserver 240 analyzes the information from the various subsystems (e.g.,camera subsystem 202, LiDAR subsystem 204, weight subsystem 206, etc.)and determines which person in a space removed which items from thespace. As discussed previously, these subsystems group information intotime windows 332 that are aligned across the subsystems. By groupinginformation into aligned time windows 332, central server 240 can findrelationships between information from disparate subsystems and gleanadditional information (e.g., which person removed which item 130). Insome embodiments, central server 240 also charges people for items theyremoved from the space when those people exit store 100.

FIGS. 6A and 6B show an example operation of central server 240. As seenin FIG. 6A, central server 240 receives information from various serversduring particular time windows. In the example of FIG. 6A, centralserver 240 receives the physical position of two people in the spacefrom camera server 225 during a first time window 332A. This disclosureuses capital ‘X’ and capital ‘Y’ to denote the physical coordinates 602of a person or object in the space and to distinguish the physicalcoordinates 602 of the person or object in the space determined bycamera server 225 and LiDAR server 230 from the local coordinatesdetermined by other components (e.g., coordinates 322 determined bycamera clients 220 and coordinates 410 determined by LiDAR sensors 405).

According to the camera server 225, the first person is at a physicalcoordinate 602 (X₁, Y₁), and the second person is at a physicalcoordinate 602 (X₂, Y₂). Additionally, central server 240 receives fromLiDAR server 230 the physical location of the two people. According tothe LiDAR server 230, the first person is at coordinate 602 (X₇, Y₇) andthe second person is at coordinate 602 (X₈, Y₈). Furthermore, centralserver 240 also receives information from weight server 235 during thefirst time window 332A. According to weight server 235, no items 130were taken during the first time window 332A.

This disclosure contemplates central server 240 using any suitableprocess for analyzing the physical position of people from camera server225 and LiDAR server 230. Although the coordinates 602 provided bycamera server 225 and LiDAR server 230 may differ from each other,central server 240 may use any appropriate process for reconciling thesedifferences. For example, central server 240 may use the coordinates 602provided by camera server 225 if the coordinates 602 provided by LiDARserver 230 do not differ from the coordinates 602 provided by cameraserver 225 by an amount that exceeds a threshold. In this manner, thecoordinates 602 provided by LiDAR sever 230 act as a check on thecoordinates 602 provided by camera server 225.

During a second time window 332B, central server 240 receives fromcamera server 225 the physical coordinates 602 of the two people.According to camera server 225, during the second time window 332B, thefirst person was at coordinate 602 (X₃,Y₃) and the second person was atcoordinate 602 (X₄, Y₄). During the second time window 332B, cameraserver 240 also receives the physical coordinates 602 of the two peoplefrom LiDAR server 230. According to the LiDAR server 230, the firstperson is at coordinate 602 (X₉, Y₉) and the second person is atcoordinate 602 (X₁₀, Y₁₀) during the second time window 332B.Additionally, central server 240 learns from weight server 235 that noitems 130 were taken during the second time window 332B.

During a third time window 332C, camera server 240 receives the physicalcoordinates 602 of the two people from camera server 225. According tothe camera server 225, the first person is at coordinate 602 (X₅, Y₅)and the second person is at coordinate 602 (X₆, Y₆). Central server 240also receives the physical coordinates 602 of the two people from LiDARserver 230 during the third time window 332C. According to the LiDARserver 230, the first person is at coordinate 602 (X₁₁, Y₁₁) and thesecond person is at coordinate 602 (X₁₂, Y₁₂) during the third timewindow 332C. Additionally, central server 240 learns from weight server235 that a particular weight sensor 500 detected that two items 130 weretaken during the third time window 332C.

In response to learning that a weight sensor 500 detected that two items130 were taken, central server 240 may undergo additional analysis todetermine which person took those two items 130. Central server 240performs any suitable process for determining which person took items130. Several of these processes are disclosed in U.S. Application No.______ entitled, “Topview Object Tracking Using a Sensor Array”(attorney docket no. (090278.0180), the contents of which areincorporated by reference herein.

FIG. 6B shows central server 240 performing an example analysis todetermine which person took items 130. As seen in FIG. 6B, centralserver 240 first determines the physical coordinates 602 of the twopeople during the third time window 332C. Central server 240 determinesthat the first person was at coordinate 602 (X₅, Y₅) during the thirdtime window 332C and the second person was at coordinate 602 (X₆, Y₆)during the third time window 332C. Central server 240 also determinesthe physical location of the weight sensor 500 that detected the itemsthat were taken. In example of FIG. 6B, central server 240 determinesthat the weight sensor 500 is located at coordinate 602 (X₁₃, Y₁₃).

Central server 240 then determines the distance from each person to theweight sensor 500. Central server 240 determines that the first personis a distance 1 from the weight sensor 500 and that the second person isa distance 2 from the weight sensor 500. Central server 240 thendetermines which person was closer to the weight sensor 500. In theexample of FIG. 4B, central server 240 determines that distance 1 isless than distance 2 and, thus, the first person was closer to theweight sensor 500 than the second person. As a result, central server240 determines that the first person took the two items 130 during thethird time window 332C and that the first person should be charged forthese two items 130.

FIG. 6C illustrates an example method 600 for operating central server240. In particular embodiments, central server 240 performs the steps ofmethod 600 to determine which person in a space took an item 130.

Central server 240 begins by receiving coordinates 602 of a first personin a space during a time window 332 in step 605. In step 610, centralserver 240 receives the coordinates 602 of a second person in the spaceduring the time window 332. Central server 240 receives an indicationthat an item 130 was taken during the time window 332 in step 615. Inresponse to reeving that indication, central server 240 analyzes theinformation to determine which person took that item 130.

In step 620, central server 240 determines that the first person wascloser to the item 130 than the second person during the time window332. Central server 240 may make this determination based on determineddistances between the people and a weight sensor 500 that detected thatthe item 130 was removed. In step 625, central server 240 determinesthat the first person took the item 130 during the time window 332 inresponse to determining that the first person was closer to the item 130than the second person. The first person may then be charged for theitem 130 when the first person exits the store 100.

Modifications, additions, or omissions may be made to method 600depicted in FIG. 6C. Method 600 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as central server 240 performing the steps, anysuitable component of tracking system 132 may perform one or more stepsof the method.

VI. Hardware

FIG. 7 illustrates an example computer 700 used in tracking system 132.Generally, computer 700 can be used to implement components of trackingsystem 132. For example, computer 700 can be used to implement a cameraclient 220, a camera server 225, a LiDAR server 230, a weight server235, and/or a central server 240. As seen in FIG. 7, computer 700includes various hardware components, such as a processor 705, a memory710, a graphics processor 715, input/output ports 720, a communicationinterface 725, and a bus 730. This disclosure contemplates thecomponents of computer 700 being configured to perform any of thefunctions of camera client 220, camera server 225, LiDAR server 230,weight server 235, and/or central server 240 discussed herein. Circuitboard 565 may also include certain components of computer 700.

Processor 705 is any electronic circuitry, including, but not limited tomicroprocessors, application specific integrated circuits (ASIC),application specific instruction set processor (ASIP), and/or statemachines, that communicatively couples to memory 710 and controls theoperation of computer 700. Processor 705 may be 8-bit, 16-bit, 32-bit,64-bit or of any other suitable architecture. Processor 705 may includean arithmetic logic unit (ALU) for performing arithmetic and logicoperations, processor registers that supply operands to the ALU andstore the results of ALU operations, and a control unit that fetchesinstructions from memory and executes them by directing the coordinatedoperations of the ALU, registers and other components. Processor 705 mayinclude other hardware that operates software to control and processinformation. Processor 705 executes software stored on memory to performany of the functions described herein. Processor 705 controls theoperation and administration of computer 700 by processing informationreceived from memory 710 and/or other computers 700. Processor 705 maybe a programmable logic device, a microcontroller, a microprocessor, anysuitable processing device, or any suitable combination of thepreceding. Processor 705 is not limited to a single processing deviceand may encompass multiple processing devices.

Memory 710 may store, either permanently or temporarily, data,operational software, or other information for processor 705. Memory 710may include any one or a combination of volatile or non-volatile localor remote devices suitable for storing information. For example, memory710 may include random access memory (RAM), read only memory (ROM),magnetic storage devices, optical storage devices, or any other suitableinformation storage device or a combination of these devices. Thesoftware represents any suitable set of instructions, logic, or codeembodied in a computer-readable storage medium. For example, thesoftware may be embodied in memory 710, a disk, a CD, or a flash drive.In particular embodiments, the software may include an applicationexecutable by processor 705 to perform one or more of the functionsdescribed herein.

Graphics processor 715 may be any electronic circuitry, including, butnot limited to microprocessors, application specific integrated circuits(ASIC), application specific instruction set processor (ASIP), and/orstate machines, that receives and analyzes video data. For example,graphics processor 715 may process video data to determine the propersignals to send to a display so that the display displays an appropriateimage. Graphics processor 715 may also process video data to identifycertain characteristics (e.g., people or objects) within the video.Graphics processor 715 may be a component of a video card that isinstalled in computer 700.

Input/output ports 720 allow peripheral devices to connect to computer700. Ports 720 may be any suitable ports, such as, parallel ports,serial ports, optical ports, video ports, network ports, etc. Peripheraldevices such as keyboards, mouses, joysticks, optical tracking devices,trackpads, touchpads, etc. can connect to computer 700 through ports720. Input and output signals are communicated between computer 700 andthe peripheral devices through ports 720.

Communication interface 725 includes any suitable hardware and/orsoftware to communicate over a network. For example, communicationinterface 725 may include a mode, network card, ethernetport/controller, wireless radio/controller, cellular radio/controller,and/or universal serial bus port/controller. Computer 700 may usecommunication interface 725 to communicate with other devices over acommunication network.

Bus 730 allows components of computer 700 to communicate with oneanother. Computer 700 may include a bus controller 730 that managescommunication over bus 730.

Although the present disclosure includes several embodiments, a myriadof changes, variations, alterations, transformations, and modificationsmay be suggested to one skilled in the art, and it is intended that thepresent disclosure encompass such changes, variations, alterations,transformations, and modifications as fall within the scope of theappended claims.

What is claimed is:
 1. A system comprising: a circuit board; and a rackcomprising: a vertically positioned panel defining a first opening thatextends along at least a portion of a width of the panel; a shelfcoupled to and extending away from the panel, the shelf comprising abottom surface, a front surface that extends upwards from the bottomsurface of the shelf, and a back surface that extends upwards from thebottom surface of the shelf, the back surface of the shelf coupled tothe panel, the back surface of the shelf defining a second opening, aportion of the second opening aligning with a portion of the firstopening; a first weight sensor coupled to the bottom surface of theshelf and positioned between the front surface of the shelf and the backsurface of the shelf; a second weight sensor coupled to the bottomsurface of the shelf and positioned between the front surface of theshelf and the back surface of the shelf; a first wire coupled to thefirst weight sensor and the circuit board, the first wire extending fromthe first weight sensor through the first and second openings; and asecond wire coupled to the second weight sensor and the circuit board,the second wire extending from the second weight sensor through thefirst and second openings.
 2. The system of claim 1, further comprisinga second shelf comprising a back surface that defines a third opening,the panel further defines a fourth opening that extends along at least aportion of the width of the panel, the back surface of the second shelfcoupled to the panel such that a portion of the third opening alignswith a portion of the fourth opening.
 3. The system of claim 1, furthercomprising: a third weight sensor coupled to the top surface of thebase; and a third wire coupled to the third weight sensor and thecircuit board, the third wire extending from the third weight sensorthrough the first opening.
 4. The system of claim 1, wherein the firstweight sensor comprises: a first load cell configured to produce a firstelectric current based on a force experienced by the first load cell; asecond load cell configured to produce a second electric current basedon a force experienced by the second load cell; a third load cellconfigured to produce a third electric current based on a forceexperienced by the third load cell; and a fourth load cell configured toproduce a fourth electric current based on a force experienced by thefourth load cell.
 5. The system of claim 4, wherein: the first weightsensor comprises a first corner, a second corner, a third corner, and afourth corner; the first load cell is positioned a first distance fromthe first corner; the second load cell is positioned a second distancefrom the second corner; the third load cell is positioned a thirddistance from the third corner; and the fourth load cell is positioned afourth distance from the fourth corner, the first distance, the seconddistance, the third distance, and the fourth distance are substantiallythe same.
 6. The system of claim 5, wherein the first wire comprises: athird wire coupled to the first load cell; a fourth wire coupled to thesecond load cell; a fifth wire coupled to the third load cell; and asixth wire coupled to the fourth load cell.
 7. The system of claim 1,wherein: the first weight sensor is configured to communicate, to thecircuit board through the first wire, a first signal indicating a weightwithin a first region of the shelf experienced by the first weightsensor, the first region positioned above the first weight sensor; andthe second weight sensor is configured to communicate, to the circuitboard through the second wire, a second signal indicating a weightwithin a second region of the shelf experienced by the second weightsensor, the second region positioned above the second weight sensor. 8.The system of claim 7, further comprising a weight server, the circuitboard configured to communicate, to the weight server, a third signalindicating the weight experienced by the first weight sensor.
 9. Thesystem of claim 8, wherein the circuit board is further configured to:communicate the third signal to the weight server via an ethernetconnection if the ethernet connection is established; communicate thethird signal to the weight server via a wireless connection if theethernet connection is not established and the wireless connection isestablished; and communicate the third signal to the weight server via auniversal serial bus connection if the ethernet connection and thewireless connection are not established and the universal serial busconnection is established.
 10. The system of claim 8, wherein the weightserver is configured to determine that an item in the first region ofthe shelf was removed based on the third signal.
 11. A systemcomprising: a circuit board; and a rack comprising: a verticallypositioned panel defining a first opening that extends along at least aportion of a width of the panel; a shelf coupled to and extending awayfrom the panel, the shelf defining a second opening, a portion of thesecond opening aligning with a portion of the first opening; a firstweight sensor coupled to the shelf; a second weight sensor coupled tothe shelf; a first wire coupled to the first weight sensor and thecircuit board, the first wire extending from the first weight sensorthrough the first and second openings; and a second wire coupled to thesecond weight sensor and the circuit board, the second wire extendingfrom the second weight sensor through the first and second openings. 12.The system of claim 11, further comprising a second shelf that defines athird opening, the panel further defines a fourth opening that extendsalong at least a portion of the width of the panel, the second shelfcoupled to the panel such that a portion of the third opening alignswith a portion of the fourth opening.
 13. The system of claim 11,further comprising: a third weight sensor coupled to the base; and athird wire coupled to the third weight sensor and the circuit board, thethird wire extending from the third weight sensor through the firstopening.
 14. The system of claim 11, wherein the first weight sensorcomprises: a first load cell configured to produce a first electriccurrent based on a force experienced by the first load cell; a secondload cell configured to produce a second electric current based on aforce experienced by the second load cell; a third load cell configuredto produce a third electric current based on a force experienced by thethird load cell; and a fourth load cell configured to produce a fourthelectric current based on a force experienced by the fourth load cell.15. The system of claim 14, wherein: the first weight sensor comprises afirst corner, a second corner, a third corner, and a fourth corner; thefirst load cell is positioned a first distance from first corner; thesecond load cell is positioned a second distance from the second corner;the third load cell is positioned a third distance from the thirdcorner; and the fourth load cell is positioned a fourth distance fromthe fourth corner, the first distance, the second distance, the thirddistance, and the fourth distance are substantially the same.
 16. Thesystem of claim 14, wherein the first wire comprises: a third wirecoupled to the first load cell; a fourth wire coupled to the second loadcell; a fifth wire coupled to the third load cell; and a sixth wirecoupled to the fourth load cell.
 17. The system of claim 11, wherein:the first weight sensor is configured to communicate, to the circuitboard through the first wire, a first signal indicating a weight withina first region of the shelf experienced by the first weight sensor, thefirst region positioned above the first weight sensor; and the secondweight sensor is configured to communicate, to the circuit board throughthe second wire, a second signal indicating a weight within a secondregion of the shelf experienced by the second weight sensor, the secondregion positioned above the second weight sensor.
 18. The system ofclaim 17, further comprising a weight server, the circuit boardconfigured to communicate, to the weight server, a third signalindicating the weight experienced by the first weight sensor.
 19. Thesystem of claim 18, wherein the circuit board is further configured to:communicate the third signal to the weight server via an ethernetconnection if the ethernet connection is established; communicate thethird signal to the weight server via a wireless connection if theethernet connection is not established and the wireless connection isestablished; and communicate the third signal to the weight server via auniversal serial bus connection if the ethernet connection and thewireless connection are not established and the universal serial busconnection is established.
 20. The system of claim 18, wherein theweight server is configured to determine that an item in the firstregion of the shelf was removed based on the third signal.